Display device, program, display method, and control device

ABSTRACT

A display device includes: a display unit that displays, at a remote position, a display image that includes a first image and a second image; an operation detection unit that detects an operation by a user on the display image; and a display control unit that adjusts a display mode for at least one of the first image and the second image based upon the operation detected by the operation detection unit.

TECHNICAL FIELD

The present invention relates to a display device, a program, a displaymethod and a control device.

BACKGROUND ART

There is an electronic device disclosed in the related art, which iscapable of detecting an operation, performed with respect to athree-dimensional object displayed in midair, via a capacitive touchsensor that calculates the distance between a finger and a touch panel(see PTL 1). While PTL 1 describes that the electronic device detects anoperation performed with respect to a three-dimensional object (target),full operability of operations performed with respect to the object isnot assured in the art disclosed in PTL 1.

CITATION LIST Patent Literature

PTL 1: Japanese Laid Open Patent Publication No. 2012-203737

SUMMARY OF INVENTION

According to the 1st aspect, a display device comprises: a display unitthat displays, at a remote position, a display image that includes afirst image and a second image; an operation detection unit that detectsan operation by a user on the display image; and a display control unitthat adjusts a display mode for at least one of the first image and thesecond image based upon the operation detected by the operationdetection unit.

According to the 2nd aspect, a program executed by a computer in adisplay device enables the computer to execute: processing fordisplaying a display image that includes a first image and a secondimage at a position set apart from the display device; processing fordetecting an operation by a user at the display device; and processingfor adjusting a display mode for at least one of the first image and thesecond image based upon the detected operation.

According to the 3rd aspect, a display method through which an image isdisplayed, comprises: displaying a display image that includes a firstimage and a second image at a position set apart from a display device;detecting an operation by a user at the display device; and adjusting adisplay mode for at least one of the first image and the second imagebased upon the detected operation.

According to the 4th aspect, a display device comprises: a display unitthat displays a display image together with reference informationdisplayed by the display device at a position set apart from the displaydevice by a predetermined distance; an operation detection unit thatdetects an operation by a user on the display image; an acquisition unitthat sets a detection reference near the display image and ascertains apositional relationship between the detection reference and theoperation by the user; and a control unit that executes display controlin which a display mode for the display image in relation to thereference information displayed by the display device is altered basedupon the positional relationship ascertained by the acquisition unit.

According to the 5th aspect, a control device that adjusts, based uponan operation by a user with respect to a display in a midair, thedisplay, comprises: an acquisition unit that ascertains a positionalrelationship between a first reference used for detection of theoperation and a position at which the operation is detected; and acontrol unit that adjusts the display in relation to a second referenceused as a depth cue for the display based upon the positionalrelationship ascertained by the acquisition unit.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 Illustrations of the structure of the display device in a firstembodiment, in an exploded perspective view in (a), in a sectional viewin (b) and in a sectional view of an example in which anotherimage-forming optical system is used in (c)

FIG. 2 A block diagram showing the essential configuration of thedisplay device in the first embodiment

FIG. 3 A schematic presentation of a midair image displayed in the firstembodiment in a plan view in (a) and in sectional views, each indicatingthe relationship among the operation detector, the midair image and thedetection reference, in (b) and (c)

FIG. 4 A schematic presentation of a midair image displayed for purposesof calibration processing in the first embodiment in illustrationsindicating the relationship between display mode adjustment and depthperception

FIG. 5 Illustrations indicating the relationship between the displaymode adjustment and depth perception in the first embodiment

FIG. 6 Illustrations each indicating the relationship between displaymode adjustment and depth perception in the first embodiment

FIG. 7 Illustrations of the calibration processing executed in the firstembodiment in sectional views, each indicating the relationship amongthe operation detector, the midair image, the detection reference andthe finger position in (a), (b), (c) and (d)

FIG. 8 Schematic illustrations each indicating the extent and directionof displacement of an icon display position occurring during thecalibration processing

FIG. 9 A schematic presentation of a midair image brought up on displayin a midair image operation mode following the calibration processing

FIG. 10 A flowchart of the calibration processing executed in a firstcalibration processing mode in the first embodiment

FIG. 11 An illustration indicating the positional relationship among themidair image, the detection reference and the furthest reach position,to which the finger reaches, that may be observed in a secondcalibration processing mode in the first embodiment

FIG. 12 A flowchart of the calibration processing executed in a secondcalibration processing mode in the first embodiment

FIG. 13 Schematic illustrations each showing how the icon display modemay be adjusted in variation 1 of the first embodiment

FIG. 14 Schematic illustrations each showing how the icon display modemay be adjusted in variation 1 of the first embodiment

FIG. 15 Schematic illustrations each showing how the icon display modemay be adjusted in variation 1 of the first embodiment

FIG. 16 A perspective presenting an external view of the display devicein variation 2 of the first embodiment

FIG. 17 Illustrations each indicating the relationship between a midairimage and index marks present at the frame member in the display deviceas observed in variation 2 of the first embodiment

FIG. 18 Illustrations of another display device in variation 2 of thefirst embodiment, in a perspective showing the external appearance ofthe display device in (a) and in a block diagram showing the essentialconfiguration of the display device in (b)

FIG. 19 A block diagram showing the essential configuration of thedisplay device in variation 3 of the first embodiment

FIG. 20 Illustrations of the calibration processing executed invariation 3 of the first embodiment in sectional views, each indicatingthe relationship among the operation detector, the midair image, thedetection reference and the finger position in (a) and (b)

FIG. 21 A flowchart of the calibration processing executed in the firstcalibration processing mode in variation 3 of the first embodiment

FIG. 22 A schematic illustration of a midair image brought up on displayin the first calibration processing mode in variation 5 of the firstembodiment

FIG. 23 Illustrations of the calibration processing executed invariation 5 of the first embodiment, in sectional views, each indicatingthe relationship among the operation detector, the midair image, thedetection reference and the finger position in (a), (b) and (c)

FIG. 24 A flowchart of the calibration processing executed in the firstcalibration processing mode in variation 4 of the first embodiment

FIG. 25 A block diagram showing the essential configuration of thedisplay device in variation 8 of the first embodiment

FIG. 26 Illustrations of the display device in variation 9 of the firstembodiment, in a perspective showing the external appearance of thedisplay device presented in (a) and in a block diagram showing theessential configuration of the display device in (b)

FIG. 27 A sectional view of the internal structure in the display devicein variation 9 of the first embodiment

FIG. 28 Illustrations each presenting an example of a midair image thatmay be brought up on display via the display device 1 in a secondembodiment, with an initial display shown in (a) and midair images, eachbrought up on display after the display mode is adjusted, shown in (b)and (c)

FIG. 29 Illustrations each presenting an example of a midair image thatmay be brought up on display via the display device 1 in the secondembodiment, with an initial display shown in (a) and midair images, eachbrought up on display after the display mode is adjusted, shown in (b)and (c)

FIG. 30 A flowchart of the calibration processing executed in the secondembodiment

FIG. 31 A flowchart of the calibration processing executed in the secondembodiment

FIG. 32 A flowchart of the calibration processing executed in the secondembodiment

FIG. 33 A flowchart of the calibration processing executed in a thirdembodiment

FIG. 34 Illustrations each presenting an example of a midair image thatmay be brought up on display via the display device in variation 3 ofthe third embodiment, with an initial display shown in (a) and midairimages, each brought up on display after the display mode is adjusted,shown in (b) and (c)

FIG. 35 A flowchart of the calibration processing executed in variation3 of the third embodiment

FIG. 36 A flowchart of the calibration processing executed in variation4 of the third embodiment

FIG. 37 A perspective presenting an exploded view of the display devicein a fourth embodiment

FIG. 38 A block diagram of the essential configuration of in the displaydevice in the fourth embodiment

FIG. 39 A flowchart of the calibration processing executed in the fourthembodiment

FIG. 40 A block diagram of the essential configuration of the displaydevice in variation 1 of the fourth embodiment

FIG. 41 A sectional view schematically illustrating the structuresadopted in a display unit and an image-forming optical system in anotherexample of variation 1 of the fourth embodiment

FIG. 42 A flowchart of the calibration processing executed in variation2 of the fourth embodiment

DESCRIPTION OF EMBODIMENTS First Embodiment

In reference to drawings, the display device in the first embodimentwill be described. The first embodiment will be described in referenceto an example in which the display device in the embodiment is mountedin an operation panel. It is to be noted that the display device in theembodiment may be mounted in an electronic apparatus other than anoperation panel. It may be mounted in, for instance, a portabletelephone, a television set, a tablet terminal, a portable informationterminal device such as a wristwatch-type terminal, a personal computer,a music player, a land-line telephone unit or a wearable device. Inaddition, the display device in the embodiment may be integrated into anelectronic system such as a digital signage system. Examples of suchdigital signage systems include a compact display unit built into, forinstance, an automatic vending machine or the like or a large displayunit, assuming a size greater than a typical adult person, which may beinstalled at a wall surface in a building. Furthermore, the displaydevice in the embodiment may be built into, for instance, a panel of anautomatic cash machine (ATM) at which the user enters a PIN number, anamount of money and the like on, a panel of an automatic ticket vendingmachine that dispenses railway tickets, bus tickets, commuter passes andthe like, or a panel on any of various types of information searchterminal systems installed in libraries, art galleries and the like.Moreover, the display device in the embodiment may be installed in anyof various types of robots (including, for instance, mobile robots andelectronic devices such as self-propelled vacuum machines).

FIG. 1(a) is an exploded perspective view of a display device 1 and FIG.1(b) shows part of the display device 1 in an enlarged side elevation.It is to be noted that for purposes of better clarity, the explanationwill be given in reference to a coordinate system assuming an X axis, aY axis and a Z axis set relative to the display device 1, as indicatedin the figure. It is to be also noted that the coordinate system set forthese purposes does not need to be an orthogonal coordinate systemassuming the X axis, the Y axis and the Z axis, and it may instead be apolar coordinate system or a cylindrical coordinate system. In otherwords, any of these coordinate systems may be set relative to thedisplay device 1, as long as the X axis is set to extend along theshorter sides of the rectangular display area of the display device 1,the Y axis is set to extend along the longer sides of the rectangulardisplay area of the display device 1 and the Z axis is set to extendalong a direction perpendicular to the display area (i.e., along theoptical axis of an image-forming optical system, to be explained later).

The display device 1 includes a body 10 having installed therein acontrol unit 20, a display unit 11, an image-forming optical system 12and an operation detector 13. The display unit 11, the image-formingoptical system 12 and the operation detector 13 are disposed inside thebody 10. The display unit 11, constituted with, for instance, a liquidcrystal display or an organic EL display, includes a plurality ofdisplay pixel arrays arranged in a two-dimensional pattern. Undercontrol executed by the control unit 20, the display unit 11 displays animage corresponding to display image data. The image-forming opticalsystem 12 is disposed further upward relative to the display unit 11 (onthe + side along the Z direction) over a predetermined distance from thedisplay unit 11. The image-forming optical system 12 may be configuredby, for instance, layering two micro-lens arrays, each having convexmicro-lenses 121 arranged in a two-dimensional array, along the Zdirection as clearly indicated in FIG. 1(b).

The image-forming optical system 12 forms a midair image or floatingimage 30 of a display image brought up on display at the display unit 11in the space above the display device 1. Namely, an image brought up ondisplay at the display unit 11 can be viewed by the user of the displaydevice 1 as the midair image 30, floating above the display device 1. Itis to be noted that the following explanation will be provided byassuming that the user is located toward the + side along the Zdirection relative to the display device 1. The midair image 30 includesa plurality of icons 30A (operation buttons) corresponding to operationbuttons via which various settings may be selected for the displaydevice 1 and instructions for implementing various functions of thedisplay device 1 may be issued. The icons 30A in the embodiment may beset over, for instance, one row by thee columns. It is to be noted thatinstead of the micro-lens arrays, pinhole arrays or slit arrays may beused to configure the image-forming optical system.

It is to be noted that the display device 1 may include an image-formingoptical system 12 other than that shown in FIGS. 1(a) and 1(b). Thedisplay device 1 may instead include, for instance, an image-formingoptical system 12A shown in FIG. 1(c), having a plurality of micromirrorelements disposed in a two-dimensional pattern therein, which isdisposed at a predetermined angle relative to the XY plane, e.g., with a45° tilt. In the image-forming optical system 12A, light having departedan image displayed at the display unit 11 and advancing toward the +side along the Z direction, is reflected toward the Y direction+ sideand forms a midair image 30, produced from a display image, at a planeranging parallel to the ZX plane. At this time, the midair image 30indicated by the dotted line in FIG. 1(c) is formed at a positionsymmetrical to the display surface of the display unit 11 relative tothe image-forming optical system 12A. In other words, a distance dbetween the display surface of the display unit 11 and the image-formingoptical system 12A matches a distance d between the image-formingoptical system 12A and the midair image 30. It is to be noted that aspecific structure that may be adopted in such an image-forming opticalsystem 9 is disclosed in, for instance, Japanese Laid Open PatentPublication No. 2016-14777. In addition, instead of the structuredescribed above, the image-forming optical system 12A may include, forinstance, convex lenses. It is to be noted that a structure thatincludes convex lenses may assume a significant thickness along theoptical axis depending upon the focal length and accordingly, animage-forming optical system that includes Fresnel lenses may be adoptedinstead. An optimal image-forming optical system 12 or 12A to beincluded in the display device 1 may be selected in correspondence to aspecific set of requirements such as those described above.

Furthermore, the display device 1 may display a midair image 30 inconjunction with an image-forming optical system 12A that adopts thelight-field method of the known art, as will be explained later inreference to variation 1 of a fourth embodiment.

In addition, the position at which the midair image 30 is displayed canbe moved along the Y direction by allowing the distance d between thedisplay unit 11 and the image-forming optical system 12A in FIG. 1(c)measured along the Z direction, to be variable. For instance, as thedistance between the display unit 11 and the image-forming opticalsystem 12A is reduced, i.e., as the display unit 11 is moved closer tothe image-forming optical system 12A, the midair image 30 is moved anddisplayed at a position further away from the user (toward the Ydirection − side). If, on the other hand, the distance between thedisplay unit 11 and the image-forming optical system 12A is increased,i.e., if the display unit 11 is moved further away from theimage-forming optical system 12A, the midair image 30 is moved anddisplayed at a position closer to the user (toward the Y direction+side). The display unit 11 can be moved along the Z direction via adrive device such as a motor, another type of actuator or the like (notshown).

It is to be noted that in conjunction with a certain type ofimage-forming optical system 12A, the relationship between the directionalong which the midair image 30 moves and the direction along which theimage-forming optical system 12A moves may be reversed from thatdescribed above. Namely, as the display unit 11 is moved closer to theimage-forming optical system 12A, the midair image 30 may be moved anddisplayed at a position closer to the user (toward the Y direction+side). If, on the other hand, the distance between the display unit 11and the image-forming optical system 12A is increased, i.e., if thedisplay unit 11 is moved further away from the image-forming opticalsystem 12A, the midair image 30 may be moved and displayed at a positionfurther away from the user (toward the Y direction − side). This meansthat the direction along which the display unit 11 is moved will beadjusted in correspondence to the type of image-forming optical system12A being used.

The operation detector 13, disposed further upward (on the + side alongthe Z direction) relative to the image-forming optical system 12, may beconfigured with, for instance, a transparent capacitive panel (hereafterwill be referred to as a capacitive panel) of the known art. Theoperation detector 13 configured with a capacitive panel forms anelectric field with electrodes constituted of a substantiallytransparent material. When the user moves his finger or a stylus towardthe midair image 30 in order to perform an operation at the displayposition of the midair image 30, the operation detector 13 detects theposition of the finger or the stylus as an electrostatic capacitancevalue. For instance, it may compare the electrostatic capacitance valuesdetected at the four corners of the transparent capacitive panel so asto detect the position of the user's finger along the X axis and the Yaxis based upon the electrostatic capacitance values detected at thefour corners.

In addition, the operation detector 13 takes an electrostaticcapacitance detection range, which is a predetermined range set upwardrelative to itself, as will be described in detail later. The operationdetector 13 detects the distance between the operation detector 13 andthe finger or the stylus within the predetermined detection range (i.e.,the position on the Z axis) based upon the electrostatic capacitancevalues detected at the four corners of the transparent capacitive panelby, for instance, comparing the electrostatic capacitance valuesdetected at the four corners. The midair image 30 must be formed via theimage-forming optical system 12 so that it occupies a position withinthe predetermined detection range of the operation detector 13, andpreferably a position around the midway point of the predetermineddetection range along the up/down direction. As described above, theoperation detector 13 detects a user operation performed at the displayposition of the midair image 30 with his finger or with a stylus,enabling the user to perform operations with respect to the midair image30 without having to directly touch the operation detector 13. It is tobe noted that while the following description will be provided byassuming that the user uses his finger to perform an operation at thedisplay position of the midair image 30, the same principle will applyto an operation performed via a stylus or the like.

FIG. 2 is a block diagram showing the control unit 20, and the displayunit 11 and the operation detector 13 controlled by the control unit 20,among the components configuring the display device 1. The control unit20, comprising a CPU, a ROM, a RAM and the like, includes an arithmeticoperation circuit that controls various structural elements of thedisplay device 1, including the display unit 11 and the operationdetector 13, based upon a control program and executes various types ofdata processing. The control unit 20 includes an image generation unit201, a display control unit 202, a calibration unit 203, a detectionreference control unit 204 and a storage unit 205. The storage unit 205includes a nonvolatile memory where the control program is stored, astorage medium where image data to be displayed at the display unit 11and the like are stored, and the like. It is to be noted that thecorrespondence between the distance from the surface of the operationdetector 13 to the fingertip and the electrostatic capacitancemanifesting when the operation detector 13 detects the fingertip isstored in advance in the storage unit 205. Accordingly, as the fingertipis positioned within the predetermined detection range of the operationdetector 13, the operation detector 13 is able to detect theelectrostatic capacitance at the fingertip, and thus detect the positionof the fingertip along the Z direction based upon the detectedelectrostatic capacitance and the correspondence stored in the storageunit 205, as explained earlier.

Based upon image data stored in the storage medium, the image generationunit 201 generates display image data corresponding to a display imageto be brought up on display at the display unit 11. The display controlunit 202 brings up the image corresponding to the display image datagenerated by the image generation unit 201 at the display unit 11. Inaddition, as the user performs an operation at the display position ofan icon 30A in the midair image 30, the display control unit 202executes display image switchover control for the display unit 11 incorrespondence to the type of the icon 30A that has been operated. It isto be noted that in response to a user operation performed at thedisplay position of the icon 30A in the midair image 30, the displaycontrol unit 202 may execute control other than the display imageswitchover control for the display unit 11. For instance, assuming thatthe user performs an operation at the display position of an icon 30A inthe midair image 30 while a movie image is on display as the displayimage at the display unit 11, the display control unit 202 may executecontrol for playing the movie image currently displayed at the displayunit 11 or for stopping the movie playback.

The calibration unit 203 executes calibration processing in first andsecond calibration processing modes, as will be described in detaillater. The detection reference control unit 204 sets a detection plane,i.e., a detection reference, in the space above the display device 1.The detection reference is a first reference used to detect an operationperformed by the user with respect to a midair image. More specifically,the detection reference control unit 204 sets the detection reference atthe position taken by (or at a position within a predetermined rangefrom) the midair image 30 within the predetermined detection range ofthe operation detector 13. The detection reference control unit 204 alsodecides that the user's finger has reached the detection reference basedupon an electrostatic capacitance value detected by the operationdetector 13. Namely, the detection reference control unit 204 decidesthat the user has performed an operation at the display position of theicon 30A when the finger position (the position on the X axis, the Yaxis and the Z axis) corresponding to the value of the electrostaticcapacitance detected by the operation detection unit 13 matches theposition of the detection reference having been set. The detectionreference control unit 204 sets the detection reference at apredetermined specific initial position. The initial position set forthe detection reference is stored in advance in the storage unit 205. Itis to be noted that the initial position of the detection reference maybe a universal position shared by all users, or a different initialposition may be set for each user based upon the history of use of thedisplay device 1 by the particular user. It is to be noted that thedetection reference control unit 204 may adjust or correct the positionof the detection reference based upon the results of calibrationprocessing to be described later.

Furthermore, the position of the detection reference (its initialposition and an adjusted or a corrected reference position) may be setover the entire plane (over the X axis and the Y axis) of the operationdetector 13 or may be set over part of the plane. Moreover, the positionof the detection reference having been set when the display device 1 waslast used, stored in the storage unit 205, may be read out and set asthe initial position of the detection reference position. It is to benoted that the detection reference control unit 204 may decide that theuser has performed an operation at the display position of the icon 30Awhen the finger position corresponding to the electrostatic capacitancevalue detected by the operation detector 13 substantially matches theposition of the detection reference, as well as when the finger positionexactly matches the position of the detection reference. A specificrange over which the finger position is considered to substantiallymatch the position of the detection reference may be set in advance.

FIG. 3(a) presents an example of a midair image 30 that may be displayedby the display device 1 and FIG. 3(b) schematically illustrates thepositional relationship among the body 10 or the operation detector 13,the midair image 30 and a detection reference 40. The midair image 30 inFIG. 3(a) includes three icons 30A set over one row by three columns, asexplained earlier, and reference lines 310 through 314 used as a depthcue (reference information). Namely, the midair image 30 includes theicons 30A constituting a first portion and the reference lines 310through 314 constituting a second portion different from the firstportion. It is to be noted that the reference lines 310 through 314 usedas a depth cue, will be described in detail later. The detectionreference 40 in FIG. 3(b) is set by the detection reference control unit204 in the vicinity of the position taken by the midair image 30, andmore specifically, at a position slightly above the midair image 30 inthe example presented in the figure. The icons within the midair image30 are indicated by bold dotted lines 30A in FIG. 3(b). It is to benoted that while the icons 30A, which are part of the midair image 30,take up positions assuming a height matching that of the midair image30, the bold dotted lines indicating the icons 30A are offset from thesolid line indicating the midair image 30 in FIG. 3(b) so as todistinguish the icons 30A.

The following is an explanation of depth cues. An image projected ontothe retina of, for instance, a human being, is a two-dimensional planarimage. However, human beings and the like are capable of perceiving athree-dimensional world by using various cues to perceive depth in realspace. Depth perception cues can be classified into two primarycategories; monocular depth cues and binocular depth cues. A monoculardepth cue, which will be described in further detail later, is a cueused by a human being or the like to ascertain the depthwise position ofa target object such as the size of another object apart from the targetobject (target), an overlap of different objects, or a reference linesuch as those mentioned earlier drawn by using the laws of perspective.Binocular depth cues include binocular parallax. Binocular parallax willbe explained as follows. When a single target is viewed with both eyes,images of the target are projected with a slight offset onto the retinasof the left and right eyes. Binocular parallax is the term used to referto this offset. Human beings and the like perceive the depthwiseposition of the target based upon the extent of the offset.

A human being or the like ascertains the depthwise position of a targetby using depth cues, examples of which are listed above. In other words,as the target itself is altered or depth cues are altered, a human beingor the like senses that the target is located at a depthwise positiondifferent from the actual position. In the method explained below, thetarget (e.g., an icon 30A) is changed relative to a monocular depth cue(e.g., the reference lines 310 through 314) so as to cause the user toperceive that the depthwise position of the target (the icon 30A) haschanged.

In FIG. 3(b), the midair image 30 is formed above the operation detector13 in the display device 1, at a position set apart from the operationdetector 13 by a distance H1, whereas the detection reference 40 is setat a position further upward relative to the operation detector 13, setapart from the operation detector 13 by a distance H2 (H1<H2). Asexplained earlier, the operation detector 13 has an electrostaticcapacitance detection range 13A set above its surface. In FIG. 3(b), theelectrostatic capacitance detection limit above the operation detector13 is indicated with a dotted line 13 a, and the interval between theelectrostatic capacitance detection limit 13 a and the operationdetector 13 is indicated as an electrostatic capacitance detection range13A. The midair image 30 and the detection reference 40 are set so as totake positions within the electrostatic capacitance detection range 13A.It is to be noted that while the detection reference 40 in FIG. 3(b) isset above the midair image 30, it may instead be set further downwardrelative to the midair image 30 or may be set to exactly match theposition of the midair image 30, as long as it takes a position withinthe electrostatic capacitance detection range 13A of the operationdetector 13. A range outside the area set for the detection reference 40within the detection range 13A will hereafter be referred to as adetection reference outside range 41.

It is to be noted that the detection reference control unit 204 mayadjust the position of the detection reference 40 described above byallowing it to move along the Z direction within the detection range13A. For instance, the detection reference control unit 204 may move thedetection reference 40, set as shown in FIG. 3(b), toward the + side orthe − side along the Z direction based upon the results of calibrationprocessing executed as will be described later. While the midair image30 and the detection reference 40 are shown in FIG. 3(b) as flat planesranging parallel to the XY plane, they do not need to be flat planes butinstead may be curved planes. In addition, the detection reference 40may include stages, each corresponding to one of the icons 30A, asindicated in FIG. 3(c), instead of being formed as a flat plane. Inother words, the distance between a given icon 30A and the part of thedetection reference 40 corresponding to the particular icon may bedifferent from the distance between another icon 30A and the part of thedetection reference 40 corresponding to the other icon. Forming stagesin the detection reference 40, as described above, is particularlyeffective when the midair image 30 is a stereoscopic image and thepositions of the plurality of icons 30A are offset relative to oneanother along the Z direction, i.e., along the up/down direction. Forinstance, the positions of detection references 40, each correspondingto one of the icons 30A, may be offset in correspondence to the offsetwith which the plurality of icons 30A in the stereoscopic midair image30 are shifted along the up/down direction. Through these measures, itis ensured that the distances between the icons 30A and thecorresponding detection references 40 remain constant. In addition, theposition of a detection reference 40 among the detection references 40each set in correspondence to one of the plurality of icons 30A shown inFIG. 3(c), may be adjusted by moving it independently. Namely, whencalibration processing has been executed based upon an operationperformed for the icon 30A located at the left end in the drawing sheeton which FIG. 3 is presented, the detection reference control unit 204may move the position of the detection reference 40 set incorrespondence to the icon 30A at the left end of the drawing along theZ direction. At this time, the detection reference control unit 204 doesnot change the Z-direction positions of the detection references 40 setin correspondence to the other icons 30A (the icons 30A at the centerand at the right end in the drawing in FIG. 3(c)).

When the user's fingertip has reached a point set apart from theoperation detector 13 by the distance H2, the operation detector 13outputs a detection output corresponding to the distance H2. Based uponthe detection output provided by the operation detector 13, thedetection reference control unit 204 decides that the user's fingertipposition has matched the detection reference 40 and accordingly decidesthat the user has performed an operation with his fingertip at thedisplay position of an icon 30A. Through this process, the displaydevice 1 detects a user operation performed at the display position ofthe particular icon 30A in the midair image 30 and executes a functioncorresponding to the icon 30A having been operated. For instance, itexecutes display image switchover at the display unit 11.

The icons 30A take positions set apart from the operation detector 13 bythe distance H1. The icons 30A are displayed as part of the midair image30 and for this reason, the visual perception of the display positionsof the icons 30A in the midair image 30, i.e., their height H1, of oneuser may be different from that of another user. In addition, the visualperception of the display positions of the icons 30A of a given user maychange depending upon the environment in which he operates the displaydevice 1. For instance, when the detection reference 40 is set so as toalign with the position of the midair image 30, a user may move hisfinger toward an icon 30A in the midair image 30 in order to perform anoperation at the display position of the particular icon 30A. In thissituation, the user may feel that there is still some distance betweenhis finger and the icon 30A although the finger has, in fact, reachedthe icon, i.e., the detection reference 40. Under such circumstances, anunintended icon operation will be executed. Another user may move hisfinger toward an icon 30A in the midair image 30 in order to perform anicon operation, and may feel that his finger has reached the icon 30Aand he is therefore, performing an operation at the display position ofthe icon 30A, though his finger is actually still set apart from theicon 30A, i.e., the detection reference 40. In this case, no iconoperation will be executed, contrary to the user's intention. In eitherscenario, the user is bound to feel that the response to his efforts aticon operation is poor.

In addition to a midair image operation mode that may be set whenperforming operations for the midair image 30, as described above, acalibration processing mode can be set in the display device 1 in theembodiment, so as to reduce the above-described feeling something wrongwith the response to icon operations. The display device 1 set in thecalibration processing mode adjusts the display mode for the midairimage 30 without altering the positional relationship between the midairimage 30 and the detection reference 40 so as to create a userperception as if the display position of the midair image 30 has movedalong the Z direction (depthwise direction), i.e., the direction inwhich the optical axis of the image-forming optical system 12 extends.As will be described in detail later, the display device 1 in theembodiment alters the depthwise position of the display of the midairimage 30 by using a monocular depth cue. Namely, the display device 1set in the calibration processing mode adjusts the display of the midairimage 30 relative to a depth cue, (e.g., the reference lines 310 through314 to be explained later) used as a second reference based upon thepositional relationship between the detection reference 40 used as afirst reference and the user operation position. In other words, thedisplay device 1 executes display control, under which the display modefor the midair image 30 is altered relative to the depth cue used asreference information, based upon the positional relationshipascertained via the operation detector 13. The depth cue used as thesecond reference is a midair display different from the icons 30A.Through these measures, the display device 1 allows the user to alterthe depthwise position at which he performs an icon operation so as toensure that icon operations performed by the user will be detected atthe position of the detection reference 40. The following is a detaileddescription of the calibration processing mode. It is to be noted that adisplay device 1 such as that shown in FIG. 1(c) adjusts the displaymode for the midair image 30 so as to create a user perception as if thedisplay position of the midair image 30 has moved along the Y direction.Namely, the apparent direction of movement of the midair image 30 asperceived by the user may be described as follows. The display mode forthe midair image 30 is adjusted so as to create a user perception as ifthe midair image 30 has moved closer to the image-forming optical system12 or has moved further away from the image-forming optical system 12.

As explained earlier, first and second calibration processing modes areavailable in the display device 1 in the embodiment. In the firstcalibration processing mode, calibration processing is executed whilethe midair image operation mode is not in effect, i.e., while midairimage operation mode execution is not underway. In the secondcalibration processing mode, calibration processing is executed whilethe midair image operation mode execution, following startup of thedisplay device 1, is underway. The processing in the first and secondcalibration processing modes is executed by the calibration unit 203shown in FIG. 2.

The first or second calibration processing mode may be selected by theuser via a calibration processing mode selector operation button (notshown) located at the display device 1. The control unit 20 may selectand execute the midair image operation mode if neither the firstcalibration processing mode nor the second calibration processing modehas been selected via the calibration processing mode selector operationbutton. In addition, if the display device 1 does not have a calibrationprocessing mode selector operation button, the second calibrationprocessing mode may be a default mode. The first calibration processingmode and the second calibration processing mode will now be described inthat order. It is to be noted that the first or second calibrationprocessing mode may be selected via an icon in the midair image insteadof via an operation button.

The first calibration processing mode will be explained first. As thedisplay device 1 is started up, the user may operate the calibrationprocessing mode selector operation button to select the firstcalibration processing mode. The calibration unit 203 in FIG. 2 startsthe first calibration processing mode once the first calibrationprocessing mode has been selected in response to the user operation. Theimage generation unit 201 generates display image data, and the displayunit 11 brings up a display image to be used in calibration processingbased upon the display image data.

FIG. 4(a) is a schematic illustration presenting an example of a displayimage provided as a midair image 300 to be used in the calibrationprocessing. The midair image 300 has a rectangular outline and includesa calibration icon 300A and reference lines 310 through 314 used as amonocular depth cue. The image generation unit 201 displays a message“Touch this icon for calibration”, superimposed on the calibration icon300A. It is to be noted that the image generation unit 201 does notnecessarily have to display the message “Touch this icon forcalibration” to start calibration processing. For instance, the user,having selected the calibration processing mode, may already becognizant of a specific operation to be performed in the calibrationprocessing mode and in such a case, the image generation unit 201 doesnot display the message.

The image generation unit 201 displays the reference lines 310 through314, set in the form of a perspective. A perspective is drawn inconformance to the laws of perspective in order to produce athree-dimensional space in a two-dimensional plane, and expressesdistances with a plurality of straight lines. In the example presentedin FIG. 4, straight reference lines 311 through 314 are set so as to runfrom the four vertices of a rectangular outline 301 of the midair image300 toward the center of the rectangle and a reference line 310 defininga rectangle smaller than the outline 301 is set near the center. Namely,a pair of reference lines 311 and 313 constitute part of one of thediagonals of the rectangular outline 301 and a pair of reference lines312 and 314 constitute part of the other diagonal of the rectangularoutline 301, with the point at which these diagonals intersect eachother corresponding to the center of the rectangular outline 301.

While the reference line 310 defines a rectangle similar to the shapedefined by the outline 301 in the example presented in FIG. 4, they donot need to be similar to each other. In addition, the reference line310 does not need to define a rectangle and may instead define a circleor another polygonal shape. Furthermore, there may be no reference line310. The reference lines 311 through 314 are straight lines connectingvertices of the shape defined by the reference line 310 and the verticesof the outline 301 of the midair image 300. In a perspective drawingthat includes such reference lines 310 through 314, the vanishing point(faraway point) at which parallel lines meet each other in the laws ofperspective is located near the center of the midair image 300.Accordingly, a user looking at such a midair image 300 experiences avisual effect whereby a point moving away from the outline 301 andmoving closer to the vanishing point, i.e., the rectangular referenceline 310 set in the vicinity of the center, appears further away fromthe user. It is to be noted that while the example presented in FIG. 4includes a single vanishing point, the present invention is not limitedto this example and it may be adopted in conjunction with a perspectivedrawing having a plurality of vanishing points. In addition, while thereference lines 311 through 314 intersect at the vanishing point in theexample presented in the FIG. 4, the reference lines 311 through 314 donot necessarily need to meet at a vanishing point.

The image generation unit 201 adjusts the display mode for the icon 300Aon the perspective drawing that includes the reference lines 310 through314 drawn as described above. In more specific terms, the imagegeneration unit 201 alters the size and the display position of the icon300A. As the size of the icon 300A is reduced and its display positionis altered along a direction running toward the vanishing point, theuser experiences a perception as if the icon 300A has moved further awayfrom the user. As the size of the icon 300A is increased and its displayposition is altered along a direction running away from the vanishingpoint, the user experiences a perception as if the icon 300A has movedcloser to the user. Thus, the reference lines 310 through 314 functionas a monocular depth cue with respect to the icon 300A.

It is to be noted that the image generation unit 201 may adjust onlyeither the size or the display position of the icon 300A.

In reference to FIG. 4(a) through FIG. 4(c), the display mode for theicon 300A and the depth perceived by the user will be explained. In FIG.4(a) through FIG. 4(c), the icon 300A is displayed toward the Ydirection − side relative to the reference line 310 inside the outline301. In the initial display provided prior to the calibrationprocessing, the icon 300A is displayed in the size and at the positionshown in FIG. 4(a). In the example presented in FIG. 4(b), the displayposition of an icon 300A1 is altered along a direction toward the Ydirection+ side and the size of the icon 300A1 is reduced relative tothose in the initial display shown in FIG. 4(a). Namely, the displayposition of the icon 300A1 in FIG. 4(b) is moved closer to the referenceline 310 along the reference lines 312 and 313 and the size of the icon300A1 is reduced based upon the extent by which the display position haschanged along the reference lines 312 and 313, i.e., based upon theextent by which the display position has moved, in comparison to thoseof the icon 300A in the initial display. As the display mode for theicon 300A in the initial display shown in FIG. 4(a) is adjusted to thatin FIG. 4(d), the user experiences a perception as if the icon 300A hasmoved further away from the user (toward the Z direction − side) basedupon its positional relationship to the reference lines 310 through 314.

In addition, the example presented in FIG. 4(c), the display position ofan icon 300A2 is altered along a direction toward the Y direction − sideand the size of the icon 300A2 is increased relative to those in theinitial display shown in FIG. 4(a). Namely, the display position of theicon 300A2 in FIG. 4(c) is altered along a direction opposite thatrunning toward the reference line 310, i.e., further away from thereference line 310, along the reference lines 312 and 313. Furthermore,the size of the icon 300A2 is increased based upon the extent by whichthe display position has changed along the reference lines 312 and 313.As the display mode for the icon 300A in the initial display shown inFIG. 4(a) is adjusted to that in FIG. 4(c), the user experiences aperception as if the icon 300A2 has moved closer to the user (toward theZ direction+ side) based upon its positional relationship to thereference lines 310 through 314.

FIG. 5 presents another example of a perception effect on the userachieved by altering the display position of an icon 300A. The size ofthe icon 300A in a midair image 300 in the initial state shown in FIG. 5is smaller than the size of the icon 300A in FIG. 4. It is to be notedthat FIG. 5 does not include a message superimposed on the icon 300A.While FIG. 5(a) shows the icon 300A in an initial display, FIG. 5(b)shows display of an icon 300A1 after the display mode is adjusted so asto create a user perception as if it has moved further away from theuser. It is to be noted that while FIG. 5(b) includes a dotted lineindicating the icon 300A in the initial display in order to facilitatethe explanation, such a dotted line is not included in the actualdisplay. In the example presented in FIG. 5, the icon 300A in theinitial display in FIG. 5(a) takes a position on the reference line 313extending toward the vanishing point. While the icon 300A1 in FIG. 5(b)assumes a size matching that of the icon 300A in the initial display andtakes a position on the reference line 313, it has moved closer to thereference line 310 or the vanishing point relative to the icon 300A inthe initial display. As a result, the user experiences a perception asif the icon 300A1 has moved further away from the user in comparison tothe icon 300A in the initial display.

FIG. 5(c) shows display of an icon 300A2 after the display mode isadjusted so as to create a user perception as if it has moved closer tothe user. It is to be noted that while FIG. 5(c) includes a dotted lineindicating the icon 300A in the initial display in order to facilitatethe explanation, such a dotted line is not included in the actualdisplay. While the icon 300A2 in FIG. 5(c) assumes a size matching thatof the icon 300A in the initial display and takes a position on thereference line 313, it has moved further away from the reference line310 or the vanishing point relative to the icon 300A in the initialdisplay. As a result, the user experiences a perception as if the icon300A2 has moved closer to the user in comparison to the icon 300A in theinitial display.

It is to be noted that the midair image 300, which includes the icon300A, 300A1 or 300A2, as shown in FIGS. 5(a) through 5(b) is displayedbased upon display image data generated by the image generation unit201.

While the icon 300A, 300A1 or 300A2 in FIG. 5 moves on the referenceline 313 extending toward the vanishing point, it will be obvious thatit may instead move on any of the other reference lines 311, 312 and 314extending toward the vanishing point. Furthermore, the icon 300A, 300A1or 300A2 does not need to take a position on a reference line among thereference lines 311 through 314, and instead it may take a position nearone of the reference lines 311 through 314. An icon 300A1 or 300A2taking a position near a reference line among the reference lines 311through 314 will move along the nearby reference line among thereference lines 311 through 314 relative to the icon 300A in the initialdisplay, i.e., along a direction running parallel to the nearbyreference line.

As described above, as the display position of the icon 300A is alteredso as to move closer to the vanishing point along a reference line whileits size remains unchanged, the user experiences a perception as if theicon 300A has moved further away from the user. As the display positionof the icon 300A is altered so as to move away from the vanishing pointalong a reference line while its size remains unchanged, the userexperiences a perception as if the icon 300A has moved closer to theuser. In this instance, the reference lines 310 through 314 function asa monocular depth cue with respect to the icon 300A. It will be obviousthat the size of the icon 300A may also be altered in the examplepresented in FIG. 5, in the same way as it is altered in the examplepresented in FIG. 4. Namely, the size of the icon 300A1 in FIG. 5(b) maybe reduced relative to the size of the icon 300A in the initial displayand the size of the icon 300A2 in FIG. 5(c) may be increased relative tothe size of the icon 300A in the initial display.

In addition, the icon 300A, 300A1 or 300A2 may take a position betweentwo reference lines instead of a position on or near a reference lineamong the reference lines 311 through 314.

FIGS. 6(a) through 6(c) present an example in which an icon 300A, 300A1or 300A2 takes a position between two reference lines 312 and 313. It isto be noted that FIG. 6 does not include a message superimposed on theicon 300A. While FIG. 6(a) shows the icon 300A in an initial displaybrought up at an intermediate position between the reference lines 312and 313, FIG. 6(b) shows display of the icon 300A1 after the displaymode is adjusted so as to create a user perception as if it has movedfurther away from the user. It is to be noted that while FIG. 6(b)includes a dotted line indicating the icon 300A in the initial displayin order to facilitate the explanation, such a dotted line is notincluded in the actual display. While the icon 300A1 in FIG. 6(b)assumes a size matching that of the icon 300A in the initial display andtakes a position between the reference lines 312 and 313, it has movedcloser to the vanishing point relative to the icon 300A in the initialdisplay. As a result, the user experiences a perception as if the icon300A1 has moved further away from the user in comparison to the icon300A in the initial display.

FIG. 6(c) shows display of the icon 300A2 after the display mode isadjusted so as to create a user perception as if it has moved closer tothe user. It is to be noted that while FIG. 6(c) includes a dotted lineindicating the icon 300 in the initial display in order to facilitatethe explanation, such a dotted line is not included in the actualdisplay. While the icon 300A2 in FIG. 6(c) assumes a size matching thatof the icon 300A in the initial display and takes a position between thereference lines 312 and 313, it has moved further away from thevanishing point relative to the icon 300A in the initial display. As aresult, the user experiences a perception as if the icon 300A2 has movedcloser to the user, in comparison to the icon 300A in the initialdisplay.

The icons 300A, 300A1 and 300A2 may each take a position between thereference lines 311 and 312, between the reference lines 313 and 314 orbetween the reference lines 311 and 314, instead.

In the example described above, the display position of the icon 300Aset between two reference lines is altered along a direction toward thevanishing point or along a direction away from the vanishing point whileits size remains unchanged. In this case, the user experiences aperception as if the icon 300A has moved further away from the user orhas moved closer to the user.

It will be obvious that the size of the icon 300A may be altered in theexample presented in FIG. 6, in much the same way as it is in theexample presented in FIG. 4. Namely, the size of the icon 300A1 in FIG.6(b) may be reduced relative to the size of the icon 300A2 in theinitial display and the size of the icon 300A2 in FIG. 6(c) may beincreased relative to the icon 300A in the initial display.

In reference to FIGS. 4 through 7, the position of the user's fingerrelative to the midair image 300 and the display mode for the midairimage 300, assumed during the calibration processing, will be explained.FIG. 7 presents sectional views, each schematically illustrating therelationship among the operation detector 13, the midair image 300, thedetection reference 40 and the position of a finger F.

The detection reference control unit 204 sets the detection reference 40at a position near the midair image 300, e.g., slightly above the midairimage 300, as indicated in FIG. 7(a). It will be obvious that thedetection reference control unit 204 may instead set the detectionreference 40 so as to align it with the midair image 300 or may be setit at a position slightly below the midair image 300. In this situation,the image generation unit 201 brings up on display the midair image 300shown in, for instance, FIG. 4(a), FIG. 5(a) or FIG. 6(a).

The user, following the instructions in the message superimposed on theicon 300A in the midair image 300, moves his fingertip F down toward theicon 300A, as shown in FIG. 7(a). As the fingertip F reaches theelectrostatic capacitance detection range 13A of the operation detector13 shown in FIG. 2, the operation detector 13 detects the movement ofthe user's fingertip F toward the icon 300A, i.e., the downwardmovement, as a change in the electrostatic capacitance. At this time,the control unit 20 ascertains the position at which the user operationis being performed in the detection reference outside range 41, detectedby the operation detector 13.

FIG. 7(b) shows the fingertip F having moved further down and havingreached a position indicated by a dotted line 50 slightly above thedetection reference 40. It is assumed that the user feels that hisfingertip F has reached the display position of the icon 300A, hasperformed an operation of pressing down the icon 300A, and will move thefingertip F upward by a predetermined distance. The operation detector13 detects the downward movement of the fingertip F described above,i.e., the fingertip F pressing down on the icon 300A, and the subsequentupward movement by the predetermined distance as changes in theelectrostatic capacitance. Once the operation detector 13 detects thefingertip F pressing down the icon 300A and the subsequent upwardmovement of the fingertip F by the predetermined distance as describedabove, the detection reference control unit 204 decides that anoperation has been performed at the display position of the icon 300A.It is to be noted that the furthest reach position to which the user'sfingertip F moves downward in order to press down the icon 300A for anoperation at the display position of the icon 300A before the fingertipF moves upward by the predetermined distance will be referred to as areach position. Namely, the position indicated by the dotted line 50will be referred to as the reach position.

If the reach position 50 is located closer to the user (toward the Zdirection+ side) relative to the detection reference 40, as shown inFIG. 7(b), the operation detector 13 cannot detect the fingertip F ofthe user at the detection reference 40. In this situation, the reachposition 50 must move closer to the midair image 300 (further away fromthe user), i.e., toward the Z direction − side, compared to the positionshown in FIG. 7(b), in order for the user's fingertip F to be detectedat the detection reference 40. In this embodiment, the display mode forthe icon 300A is adjusted so as to lead the user to move his fingertipF, at the reach position 50, toward the Z direction − side in comparisonto the position indicated in FIG. 7(b). Namely, the user is made toexperience a perception as if the midair image 300 and the icon 300A arelocated at the position indicated by the two-point chain line in FIG.7(b) so as to lead the user to move his fingertip F toward the Zdirection − side relative to the position indicated in the figure.During this process, the image generation unit 201 determines thedirection along which the display position of the icon 300A is to bealtered in the user's perception based upon the positional relationshipbetween the reach position 50 and the detection reference 40. If thereach position 50 is located closer to the Z direction+ side relative tothe detection reference 40, as shown in FIG. 7(b), the image generationunit 201 determines that the direction along which the display positionof the icon 300A is to be altered in the user's perception is toward theZ direction − side. The image generation unit 201 switches the displaymode for the icon 300A, currently on display as shown in FIG. 4(a), sothat its display position and size change, as shown in FIG. 4(b).Namely, the image generation unit 201 generates display image data thatinclude the icon 300A taking a display position shifted toward the Ydirection − side and a reduced size. It is to be noted that if themidair image 300 shown in FIG. 5(a) appears in the initial display, theimage generation unit 201 alters the display position of the icon 300Aso that it moves closer to the vanishing point on the reference line313, as does the icon 300A1 in FIG. 5(b). In addition, if the midairimage 300 shown in FIG. 6(a) appears in the initial display, the imagegeneration unit 201 alters the display position of the icon 300A so thatthe display position moves closer to the vanishing point in an areabetween the reference line 312 and the reference line 313, as does theicon 300A1 in FIG. 6(b). In other words, the image generation unit 201adjusts the icon 300A constituting the first portion relative to thereference lines 310 through 314 constituting the second portion.

In addition, while the display mode for the icon 300A is adjusted byaltering its display position and size in the example presented in FIG.4(a) and FIG. 4(b) and by altering the display position of the icon 300Ain the examples presented in FIGS. 5 and 6, the image generation unit201 may instead adjust the size of the icon 300A without altering itsdisplay position. In such a case, the image generation unit 201 mayreduce the size of the icon 300A, as shown in FIG. 4(d). By reducing thesize of the icon 300A relative to the reference lines 310 through 314, auser perception is created as if the display position of the icon 300Ahas moved toward the Z direction − side, as in the example presented inFIG. 4(b).

The image generation unit 201 adjusts the display mode for the icon 300Aas described above so that the user experiences a perception as if thedisplay position of the icon 300A has moved along a direction determinedbased upon an operation performed by the user to press down the icon300A. It is to be noted that the user performing a press-down operationdoes not need to move his finger strictly along the Z direction. Evenwhen the user performs a press-down operation at an angle relative tothe Z axis, if a Z direction component is included in the direction ofthe gesture, the control unit 20 may decide that the user has performeda press-down operation. In response, the image generation unit 201 mayadjust the display mode for the icon 300A so as to create a userperception as if the display position of the icon 300A has moved alongthe Z direction, i.e., the direction determined based upon thepress-down operation.

It is to be noted that the image generation unit 201 may adjust thedisplay mode for the icon 300A based upon a press-down operationperformed by the user so as to create a user perception as if thedisplay position of the icon 300A has moved along a direction matchingthat of the press-down operation performed by the user, determined basedupon the press-down operation.

The image generation unit 201 determines a displacement quantityrepresenting the extent to which the icon 300A is to move and a changequantity representing the extent to which its size is to be alteredbased upon the amount of offset between the detection reference 40 andthe reach position 50, i.e., based upon the distance between them alongthe Z direction. The correlation among the distance between thedetection reference 40 and the reach position 50, the displacementquantity representing the extent to which the icon 300A is to move andthe change quantity with respect to the size of the icon 300A isdetermined in advance based upon the results of testing and the like andis stored as correlation data in the storage unit 205. The imagegeneration unit 201 determines the displacement quantity and the changequantity for the icon 300A by referencing the correlation data andadjusts the display mode for the icon 300A accordingly.

Next, in reference FIG. 8, the displacement quantity and the directionof displacement along which the icon 300A moves will be explained. FIG.8(a), similar to FIG. 6(a), shows an icon 300A in an initial display ata display position between the reference line 312 and the reference line313. In this situation, the image generation unit 201, having read out adisplacement quantity L1 from the storage unit 205, sets a positionshifted by the displacement quantity L1 toward the vanishing point alonga direction indicated by an arrow A1, i.e., toward the Y direction −side along the Y axis, as the display position for the icon 300A.

When the icon 300A in the initial display is located between thereference lines 311 and 312, the image generation unit 201 may move thedisplay position of the icon 300A by the displacement quantity L1 towardthe X direction − side along the X axis, i.e., toward the vanishingpoint. When the icon 300A is located between the reference lines 313 and314, the image generation unit 201 may move the display position of theicon 300A by the displacement quantity L1 toward the X direction+ sidealong the X axis. When the icon 300A is located between the referencelines 311 and 314, the image generation unit 201 may move the displayposition of the icon 300A by the displacement quantity L1 toward the Ydirection+ side along the Y axis.

It is to be noted that the displacement quantity and the displacementdirection for the icon 300A indicated in FIG. 4(a) are the same as thosefor the icon 300A described above in reference to FIG. 8(a).

Next, in reference to FIG. 8(b), the displacement quantity and thedisplacement direction for an icon 300A displayed on the reference line313 will be explained. FIG. 8(b), similar to FIG. 5(a), shows the icon300A taking a position on the reference line 313. In this case, theimage generation unit 201, having read out a displacement quantity L1from the storage unit 205, sets a position shifted by the displacementquantity L1 toward the vanishing point along the reference line 313 asindicated by the arrow A2, as the display position for the icon 300A.The image generation unit 201 moves the display position for the icon300A by a displacement quantity L1X toward the X direction+ side and bya displacement quantity L1Y toward the Y direction − side.

It is to be noted that if the icon 300A takes a position on thereference line 311 in the initial display, the image generation unit 201may move the display position of the icon 300A by the displacementquantity L1 along the reference line 311. If the icon 300A takes aposition on the reference line 312 in the initial display, the imagegeneration unit 201 may move the display position of the icon 300A bythe displacement quantity L1 along the reference line 312. If the icon300A takes a position on the reference line 314 in the initial display,the image generation unit 201 may move the display position of the icon300A by the displacement quantity L1 along the reference line 314(toward the Y direction+ side and the X direction+ side).

It is to be noted that when the icon 300A takes a position near areference line among the reference lines 311, 312, 313 and 314 in theinitial display, it is moved by a predetermined displacement quantityalong a direction running parallel to the reference line among thereference lines 311, 312, 313 and 314.

In addition, while the display mode is adjusted by moving the icon 300Aalong a reference line in the description provided above, it is notstrictly necessary that the icon 300A be moved along a reference line,as long as the image generation unit 201 moves the icon 300A along adirection other than the direction running perpendicular to thereference line.

The displacement quantities along the X direction and the Y directiondetermined for the icon 300A as described above are stored into thestorage unit 205.

If the adjustment in the display mode for the icon 300A includes achange in its size, as in the example presented in FIG. 4, the imagegeneration unit 201 determines the size of the icon 300A by referencingthe correlation data stored in the storage unit 205 and stores the sizethus determined as a change quantity into the storage unit 205.

The change quantity with respect to the size of the icon 300A and thedisplacement quantity with respect to the display position of the icon300A are determined so that they increase/decrease based upon anincrease/decrease in the distance between the reach position 50 and thedetection reference 40. The change quantity with respect to the size ofthe icon 300A and the displacement quantity with respect to the displayposition of the icon 300A may be determined so that theyincrease/decrease linearly based upon an increase/decrease in thedistance between the reach position 50 and the detection reference 40 ormay instead be determined so that they increase/decrease in steps eachcorresponding to an increase/decrease by a predetermined extent in thedistance between the reach position 50 and the detection reference 40.Furthermore, the change quantity with respect to the size of the icon300A and the displacement quantity with respect to the display positionof the icon 300A may be determined so that they are each changed by apredetermined fixed value regardless of the distance between the reachposition 50 and the detection reference 40 and, in such a case, thepredetermined value may be selected by the user.

The display control unit 202 brings up on display a midair image 300that includes the icon 300A1 displayed in a display mode adjusted fromthat for the icon 300A, expressed with the display image data generatedby the image generation unit 201 as described above. As a result, thedisplay device 1 is able to create a user perception as if the icon 300Ahas moved toward the Z direction − side, i.e., away from the user. Inother words, the user experiences a perception as if the midair image300 and the icon 300A, which actually remain unmoved along the Zdirection, have moved to the position indicated by the two-point chainline in FIG. 7(b). The user, perceiving as if the icon 300A is nowdisplayed at a position further away from himself, is expected toperform an operation with respect to the icon 300A by positioning hisfingertip F further toward the Z direction − side. As a result, thereach position 50 of the user's fingertip F will move further toward theZ direction − side relative to the reach position shown in FIG. 7(b),and the reach position 50 will arrive at the detection reference 40 asindicated in FIG. 7(c). Consequently, the operation detector 13 will beable to detect the user's fingertip F at the detection reference 40 andthe control unit 30 will be able to ascertain the user operationposition detected at the detection reference 40.

In the example described above, the reach position 50 of the finger isabove (toward the Z direction+ side) relative to the detection reference40 and, as a result, the user operation cannot be detected at thedetection reference 40. When the reach position 50 is located under(toward the Z direction − side) relative to the detection reference 40and the user operation thus cannot be detected at the detectionreference 40, too, the detection reference control unit 204 determinesthe reach position 50 and the image generation unit 201 adjusts thedisplay mode for the icon 300A based upon the reach position 50 thusdetermined, in the same way as that described above. The positionalrelationship between the reach position 50 and the detection reference40 under such circumstances is illustrated in FIG. 7(d). In thissituation, the reach position 50 must move closer to the user, i.e.,closer to the Z direction+ side, compared to the position shown in FIG.7(d), in order for the user's fingertip F to be detected at thedetection reference 40. Namely, the user is made to experience aperception as if the midair image 300 and the icon 300A are located atthe position indicated by the two-point chain line in FIG. 7(d) so as tolead the user to move his fingertip F toward the Z direction+ siderelative to the position indicated in the figure. The image generationunit 201 thus determines the direction along which the display positionof the icon 300A is to be altered in the user's perception based uponthe positional relationship between the reach position 50 and thedetection reference 40. If the reach position 50 is located closer tothe Z direction − side relative to the detection reference 40, as shownin FIG. 7(d), the image generation unit 201 determines the directionalong which the display position of the icon 300A is to be altered inthe user's perception toward the Z direction+ side. The image generationunit 201 adjusts the display mode for the icon 300A, currently ondisplay as shown in FIG. 4(a), so that its display position and sizechange, as shown in FIG. 4(c). Namely, the image generation unit 201moves the display position of the icon 300A toward the Y direction+ sideand increases the size of the icon 300A. It is to be noted that if theicon 300A shown in FIG. 5(a) or FIG. 6(a) appears in the initialdisplay, the display mode for the icon 300A is adjusted as shown in FIG.5(c) or FIG. 6(c). In other words, the image generation unit 201 adjuststhe icon 300A constituting the first portion relative to the referencelines 310 through 314 constituting the second portion.

In this case, too, the image generation unit 201 determines the extentto which the icon 300A is to move and the extent to which its size is tobe altered based upon the amount of offset between the detectionreference 40 and the reach position 50, i.e., based upon the distancebetween them along the Z direction, as in the adjustment of the displaymode for the icon 300A1. The image generation unit 201 determines thedisplacement quantity and the size change quantity for the icon 300A byreferencing the correlation data, and generates display image data thatinclude the icon 300A2 with an adjusted display mode. The displacementquantity by which the icon 300A is to move along the X direction and theY direction and the change quantity indicating the extent to which thesize of the icon 300A changes thus determined are stored into thestorage unit 205. The display control unit 202 brings up on display amidair image 300 that includes the icon 300A2 displayed in a displaymode adjusted from that for the icon 300A expressed with the displayimage data generated by the image generation unit 201. As a result, thedisplay device 1 is able to create a user perception as if the icon 300Ahas moved toward the Z direction+ side, i.e., closer to the user. Theuser, perceiving as if the icon 300A is now displayed at a positioncloser to himself, is expected to perform an operation with respect tothe icon 300A by positioning his fingertip F further toward the Zdirection+ side. Consequently, the reach position 50 of the user'sfingertip F will move further toward the Z direction+ side relative tothe furthest reach position shown in FIG. 7(d), and the reach position50 will arrive at the detection reference 40 as indicated in FIG. 7(c).As a result, the operation detector 13 will thus be able to detect theuser's fingertip F at the detection reference 40.

The image generation unit 201 adjusts the display mode for the icon 300Aas described above so as to create a user perception as if the displayposition of the icon 300A has moved along a direction opposite from thedirection determined based upon the press-down operation performed bythe user as if to press down on the icon 300A. It is to be noted thatthe user performing a press-down operation does not need to move hisfinger strictly along the Z direction. Even when the user performs apress-down operation at an angle relative to the Z axis, as long as a Zdirection component is included in the direction of the gesture, thecontrol unit 20 decides that the user has performed a press-downoperation. In response, the image generation unit 201 may adjust thedisplay mode for the icon 300A so as to create a user perception as ifthe display position of the icon 300A has been altered along the Zdirection, i.e., a direction determined based upon the press-downoperation.

It is to be noted that the image generation unit 201 may adjust thedisplay mode for the icon 300A based upon a press-down operationperformed by the user so that a user perception is created as if thedisplay position of the icon 300A has been altered along a directionopposite that of the press-down operation performed by the user,determined based upon the press-down operation.

It is to be noted that when the reach position 50 is at the detectionreference 40, too, the detection reference control unit 204 determinesthe reach position 50 in much the same way as that described above.However, since the reach position 50 is at the detection reference 40,the image generation unit 201 does not adjust the display mode for theicon 300A to create a user perception as if the depthwise position ofthe icon 300A has changed.

In addition, when the reach position 50 is further downward relative tothe detection reference 40, the fingertip F passes through the detectionreference 40 before the fingertip F reaches the reach position 50. Inthis situation, the detection reference control unit 204 decides thatthe finger has reached the detection reference 40 based upon thedetection output provided by the operation detector 13. However, theimage generation unit 201 does not switch the display at the displayunit 13 in the first calibration processing mode. Likewise, when thereach position 50 is aligned with the detection reference 40, the imagegeneration unit 201 does not switch the display at the display unit 11either. It will be obvious that when the fingertip F has moved to reachthe detection reference 40, the image generation unit 201 may notify theuser that the fingertip F has reached the detection reference 40 with,for instance, a highlighted display of the icon 300A by flashing theicon 300A.

While the user presses down on the icon 300A as an operation performedat the display position of the icon 300A in the example described above,the present invention is not limited to this example. Namely, when theoperation detector 13 has detected a predetermined non-contact operationperformed by the user in relation to the icon 300A, the image generationunit 201 may adjust the display mode for the icon 300A based upon thelocation where the predetermined non-contact operation was performed,i.e., based upon the position at which the predetermined non-contactoperation has been detected by the operation detector 13. Thepredetermined non-contact operation may be performed by the user bymaking a gesture as if to touch the icon 300A. In response, the imagegeneration unit 201 may adjust the display mode for the icon 300A basedupon the position at which the user has made the gesture of touching theicon 300A. The operation performed by the user making a gesture oftouching the icon 300A may be, for instance, a gesture of swiping theicon 300A with the user's hand. In addition, the position at which theuser has performed the operation by making a gesture of touching theicon 300A may be determined based upon the position at which the user'shand, having made the swiping gesture, is determined to have becomestill or based upon the position at which the user has started makingthe swiping gesture.

In addition, the user may perform the predetermined non-contactoperation by moving his finger F downward by a distance L1, then makinga U-turn and moving it upward by the distance L1. Namely, thepredetermined non-contact operation in this instance follows a U-turntrajectory with the descending distance and the ascending distancematching each other. Furthermore, the predetermined non-contactoperation may follow a trajectory in the shape of the letter V insteadof a U. Moreover, the predetermined non-contact operation may beperformed by first moving the finger F downward by the distance L1 andthen moving it back upward by the distance L1 along the descendingtrajectory. In addition, the descending distance L1 and the ascendingdistance L1 in the predetermined non-contact operation may be differentfrom each other. Namely, the predetermined non-contact operation simplyneeds to be performed by moving the finger upward continuously after ithas been moved downward.

Furthermore, the user may perform the predetermined non-contactoperation by first moving the finger F downward by the distance L1 andthen holding the finger F still over a predetermined length of time, orby first moving the finger F downward by the distance L1 and then movingthe finger F laterally over at least a predetermined distance L2.

The predetermined non-contact operations that may be performed are notlimited to those represented by the various trajectories of the finger Fdescribed above and the user may perform a non-contact operation thatfollows another trajectory as long as the trajectory of movement (thetrajectory of the movement of the finger F or a hand) can be detected bythe operation detector 13. It is to be noted that an optimal detectionposition, corresponding to a given predetermined non-contact operation,may be set as the predetermined non-contact operation detection positionfor the operation detector 13. For instance, when the user performs thepredetermined non-contact operation by moving his finger F downward bythe distance L1, making a U-turn and moving his finger F upward by thedistance L1, the non-contact operation may be detected at the lowermostposition at which the U-turn is made. In another example, thepredetermined non-contact operation detection position may be set at apoint halfway through the distance L1.

It is to be noted that the method through which the reach position 50 isdetermined by the detection reference control unit 204 is not limited tothat described above, in which the reach position 50 is determined basedupon the shift from the downward movement to the upward movement by thepredetermined distance, and it may be determined through any of thevarious other methods to be described below. For instance, the user,perceiving that his fingertip F, having reached the display position ofthe icon 300A, has pressed down on the icon 300A, may stop moving hisfinger downward, i.e., may stop pressing down the icon. In this case,the detection reference control unit 204 may decide that the finger hasstopped pressing down when there is no longer any significant change inthe value of the electrostatic capacitance detected by the operationdetector 13, and may determine, i.e., confirm, the position at which thefinger has stopped pressing down as the reach position 50. It is to benoted that it may decide that the downward movement has stopped when thevalue of the electrostatic capacitance detected by the operationdetector 13 has remained unchanged for a short span of time of, forinstance, 0.1 sec through 1 sec. In another method, the detectionreference control unit 204 may detect the velocity vector of themovement of the user's finger, i.e., the finger movement velocity andthe finger movement direction, based upon a change in the electrostaticcapacitance. Namely, based upon a change in the electrostaticcapacitance, the detection reference control unit 204 may detect thatthe direction of the finger velocity vector has changed from thedownward direction to the opposite direction and that the velocityvector along the opposite direction has reached a predetermined leveland, accordingly, may designate the position taken by the finger whenthe velocity vector achieving the predetermined level along the oppositedirection is detected as the reach position 50. If the predeterminedlevel for the velocity vector is set substantially equal to 0, theposition taken by the finger when the direction of the velocity vectorshifts from downward to the opposite direction, i.e., the lowermostposition, will be determined to be the reach position. If, on the otherhand, the predetermined level is set to a value other than 0 in thismethod, a position taken by the finger, set apart from the lowermostposition by a predetermined distance along the upward direction isdetermined as the reach position 50. As explained above, the reachposition 50 is set at the lowermost position taken by the fingertip F asthe finger is judged by the detection reference control unit 204 to haveperformed an operation at the display position of the icon or at aposition near the lowermost position.

In addition, the detection reference control unit 204 determines thereach position in reference to the part of the finger or the stylusappearing to come in contact with the icon 300A in the midair image 300,i.e., the position of the fingertip or the position of the lowermostpart of the stylus in the example presented above. As an alternative,the detection reference control unit 204 may determine the reachposition in reference to the position of the fingernail tip of theuser's finger or in reference to the position of the first joint of thefinger. Furthermore, the icon may be operated with the user's foot orelbow instead of the user's finger, and in such a case, the detectionreference control unit 204 may determine the reach position in referenceto the foot or the elbow. When the icon operation is performed via astylus, a specific position on the stylus may be marked and thedetection reference control unit 204 may determine the reach position inreference to the marked position. It is desirable that when the reachposition is determined in reference to the position of the first jointof the finger, the position of a mark on the stylus or the like, theoperation detector 13 be configured with an image-capturing device orthe like, such as that to be described later in reference to variation9, instead of the capacitive panel.

Furthermore, while the detection reference 40 is a single plane ormultiple planes defined in stages in the description provided above, thedetection reference 40 may be formed as a zone with a predetermineddepth present between an upper plane and a lower plane. In such a case,the lower plane of the detection reference 40 may be set above (towardthe Z direction+ side) relative to the midair image 30, the upper planemay be set below (toward the Z direction − side) relative to the midairimage 30, or the detection reference 40 may be set so that the midairimage 30 is positioned between the upper plane and the lower plane. Inconjunction with this detection reference, the detection referencecontrol unit 204 is able to make an even more reliable decisionregarding the operation performed at the display position. For instance,the user's finger may move downward from a position diagonally above theicon 30A instead of a point directly above the icon 30A. In such a case,if the detection reference 40 is a planar reference such as that shownin FIG. 3, the finger may not pass through the detection reference 40directly above the icon 30A and instead may pass through an area besidethe detection reference 40. Under such circumstances, the detectionreference control unit 204 may not be able to make a decision regardingthe operation performed by the finger at the display position of theicon 30A. If, on the other hand, the detection reference 40 assumes apredetermined thickness, the detection reference control unit 204 isable to detect the finger entering the detection reference 40 with highreliability even when the finger moves downward from a point diagonallyabove. Moreover, even when the finger moves parallel to the midair image30 to perform an operation at the display position of the icon 30A, too,the detection reference control unit 204 is able to detect with highreliability that the finger has entered the detection reference 40 sinceit has a predetermined thickness.

It is to be noted that if the predetermined non-contact operation is notperformed within the detection reference 40 assuming a predeterminedthickness as described above, the detection reference control unit 204makes a decision that the predetermined non-contact operation has notbeen performed. For instance, the user may perform the predeterminednon-contact operation by first moving his finger F downward by apredetermined distance L1, then making a U-turn and moving the finger Fupward by the distance L1. However, the user, having moved his finger Fdownward by the distance L1 within the detection reference 40 may onlymove the finger F upward over a distance short of the distance L1. Insuch a case, the detection reference control unit 204 makes a decisionthat the predetermined non-contact operation has not been performed bythe user.

Based upon the results of the calibration processing executed in thefirst calibration processing mode as described above, the display modefor an icon 30A on display in the midair image operation mode as shownin FIG. 3(a) is adjusted. Namely, the image generation unit 201 adjuststhe display mode for an icon 30A on display in the midair imageoperation mode based upon the results of the calibration processing byreferencing the displacement quantity and the displacement direction forthe icon 300A stored in the storage unit 205 for and the change quantitywith respect to the size of the icon 300A. FIG. 9 shows a midair image30 on display in the midair image operation mode, which reflects theresults of the calibration processing executed in the first calibrationprocessing mode, as described above. FIG. 9(a), similar to FIG. 3(a),shows a midair image 30 in the initial display. In FIG. 9(a), the icons30A disposed over one row by three columns are assigned, starting fromthe left side in the drawing, with reference signs 30Aa, 30Ab and 30Ac.Namely, a reference line closest to the display position at which theicon 30Aa is displayed is the reference line 313, reference linesclosest to the display position at which the icon 30Ab is displayed arethe reference lines 312 and 313, and a reference line closest to thedisplay position at which the icon 30 Ac is displayed is the referenceline 312.

FIG. 9(b) shows a midair image 30 reflecting the results of calibrationprocessing executed when the reach position 50 is above the detectionreference 40 (see FIG. 7(b)). Namely, FIG. 9(b) shows a midair image 30with the display mode adjusted so that a user perception is created asif the icons 30A have moved further away from the user. It is to benoted that the icons 30A in the initial display are indicated by dottedlines in FIG. 9(b) so as to facilitate the explanation.

Icons 30A1 a, 30A1 b and 30A1 c are respectively displayed by adjustingthe display mode for the icons 30Aa, 30Ab and 30Ac in the initialdisplay. The icon 30A1 a, smaller than the icon 30Aa, is displayed at aposition moved toward the vanishing point along the reference line 313.The icon 30A1 b, smaller than the icon 30Ab, is displayed at a positionmoved toward the vanishing point along the Y axis. The icon 30A1 c,smaller than the icon 30Ac, is displayed at a position moved toward thevanishing point along the reference line 312. As a result, the userperceives as if the icons 30A1 a, 30A1 b and 30A1 c have moved furtheraway from the user in relative terms with respect to the reference lines312 and 313.

FIG. 9(c) shows a midair image 30 reflecting the results of calibrationprocessing executed when the reach position 50 is below the detectionreference 40 (see FIG. 7(d)). Namely, FIG. 9(c) shows a midair image 30with the display mode adjusted so that a user perception is created asif the icons 30A have moved closer to the user. It is to be noted thatthe icons 30A in the initial display are indicated by dotted lines inFIG. 9(c) so as to facilitate the explanation.

Icons 30A2 a, 30A2 b and 30A2 c are respectively displayed by adjustingthe display mode for the icons 30Aa, 30Ab and 30Ac in the initialdisplay. The icon 30A2 a, larger than the icon 30Aa, is displayed at aposition moved further away from the vanishing point along the referenceline 313. The icon 30A2 b, larger than the icon 30Ab, is displayed at aposition moved further away from the vanishing point along the Y axis.The icon 30A2 c, larger than the icon 30Ac, is displayed at a pointmoved further away from the vanishing point along the reference line312. As a result, the user perceives that the icons 30A1 a, 30A1 b and30A1 c have moved closer to the user in relative terms in reference tothe reference lines 312 and 313.

Through the process described above, a midair image 30 is displayed byadjusting the display mode for the icons 30A in the midair imageoperation mode based upon the distance between the user operationposition, detected in the first calibration processing mode, and thedetection reference 40.

It is to be noted that while the image generation unit 201 adjusts thedisplay mode for all of the plurality of icons 30Aa, 30Ab and 30Ac basedupon the results of the calibration processing in the example presentedin FIG. 9, the present invention is not limited to this example. Theimage generation unit 201 may adjust the display mode for at least oneof the icons 300A among the plurality of icons 30Aa through 30Ac, or itmay individually adjust the display mode for each of the icons 30Aa,30Ab and 30A.

The relationship between the first calibration processing mode describedabove and the midair image operation mode will be described in referenceto the flowchart presented in FIG. 10. After the display device 1 isstarted up, the processing in the flowchart presented in FIG. 10 isexecuted by the control unit 20 is based upon a program. The program isstored in the storage unit 205.

In step S1, the first calibration processing mode, selected by the uservia the calibration processing mode selector operation button, isrecognized as the selected mode, and then the operation proceeds to stepS2. In step S2, the calibration unit 203 shown in FIG. 2 starts thefirst calibration processing mode before the operation proceeds to stepS3.

In step S3, the image generation unit 201 generates display image datafor a calibration display image, the display control unit 202 brings upon display at the display unit 11 the calibration image based upon thedisplay image data and the detection reference control unit 204 sets thedetection reference 40 at a predetermined initial position. The displayimage at the display unit 11 is the calibration midair image 300 in FIG.4 generated via the image-forming optical system 12. The midair image300 includes the icon 300A, the reference lines 310 through 314, withthe message “Touch this icon for calibration”. In step S4, the operationdetector 13 detects a downward movement of the user's fingertip F, andthe operation proceeds to step S5.

In step S5, the detection reference control unit 204 shown in FIG. 2makes a decision, based upon the detection output provided by theoperation detector 13, as to whether or not the finger has reached thereach position 50. If an affirmative decision is made in step S5, i.e.,if it is decided that the finger has reached the reach position, theoperation proceeds to step S6. If a negative decision is made in stepS5, i.e., if it is decided that the finger has not become still, theoperation waits in standby until an affirmative decision is made. Instep S6, the detection reference control unit 204 adjusts the displaymode for the icon 300A based upon the distance between the reachposition 50 and the detection reference 40. In addition, the imagegeneration unit 201 stores data indicating the displacement quantitywith respect to the display position of the icon 300A and the changequantity with respect to the size of the icon 300A into the storage unit205 shown in FIG. 2, before the operation proceeds to step S7.

In step S7, the operation exits the first calibration processing modeand proceeds to step S8. In step S8, the midair image operation mode isstarted, and the operation then proceeds to step S9. In step S9, themidair image 30 for the midair image operation mode shown in FIG. 9,which includes icons 30A, is displayed. At this time, the imagegeneration unit 201 reads out the data indicating the displacementquantity with respect to the display position of the icon 300A and thechange quantity with respect to its size, having been stored into thestorage unit 205 in step S6, from the storage unit 205. Based upon thedata thus read out, the image generation unit 201 sets the displaypositions and the sizes of the icons 30A for the midair image operationmode. Through this process, a midair image 30, having icons 30Aoptimized for the user operation characteristics through the firstcalibration processing mode, is brought up on display in the midairimage operation mode.

As the user moves his finger down toward the midair image 30 in order toperform an operation at the display position of an icon 30A, theoperation detector 13 shown in FIG. 2 detects the downward movement ofthe user's finger in step S10, and then the operation proceeds to stepS11. In step S11, the detection reference control unit 204 makes adecision based upon the detection output provided by the operationdetector 13 as to whether or not the finger has reached the detectionreference 40. If an affirmative decision is made in step S11, i.e., ifit is decided that the finger has reached the detection reference 40,the operation proceeds to step S12. If a negative decision is made instep S11, i.e., if it is decided that the finger has not reached thedetection reference 40, the operation waits in standby until anaffirmative decision is made. In step S12, the display control unit 202switches the display image at the display unit 11 to a display imagecorresponding to the icon 30A having been operated, and then theoperation proceeds to step S13. In step S13, a decision is made as towhether or not an operation has been performed to stop or shut down thedisplay device 1. If an affirmative decision is made in step S13, i.e.,if an operation has been performed to stop the display device 1, thedisplay device 1 stops. If a negative decision is made in step S13,however, the operation returns to step S10.

While the first calibration processing mode is executed immediatelyafter the display device 1 is started up so as to precede the midairimage operation mode in the example described above, the firstcalibration processing mode may instead be executed following the midairimage operation mode. For instance, the user, having experiencedfrustration while performing an operation at the display position of anicon 30A in the midair image operation mode, may operate the calibrationprocessing mode selector operation button at the display device 1 inorder to select the first calibration processing mode. In this case, thefirst calibration processing mode is executed by interrupting the midairimage operation mode that is underway and the midair image operationmode is then resumed after the first calibration processing mode ends.It is to be noted that while the display device 1 selects the firstcalibration processing mode in response to a user operation of theoperation button in the example described above, the calibration unit203 may detect signs of annoyance experienced by the user, which islikely attributable to difficulty in performing an operation at thedisplay position of the icon 30A, and may implement the firstcalibration processing mode instead. The display device 1 may, forinstance, detect the pulse rate of the user (biometric information) anda pulse rate exceeding a predetermined value may be determined to be asign of user discomfort.

Next, the second calibration processing mode will be described inreference to FIG. 11 and FIG. 12. It is to be noted that the processingdescribed earlier in reference to the first calibration mode may also beexecuted, as appropriate, in the second calibration processing modedescribed below.

FIG. 11 illustrates a midair image 30 for the midair image operationmode, the detection reference 40 and the reach position 50 to which thefingertip F reaches, whereas FIG. 12 presents a flowchart of theoperation executed in the second calibration processing mode. Theprocessing is executed in the steps in the flowchart presented in FIG.12, by the control unit 20 based upon a program, following startup thedisplay device 1.

In step S21, the second calibration processing mode is recognized as theselected mode, and then the operation proceeds to step S22. In step S22,the midair image operation mode and the second calibration processingmode start concurrently, before the operation proceeds to step S23. Instep S23, the midair image 30 shown in FIG. 3, which includes the icons30A, is displayed and the detection reference control unit 204 in FIG. 2sets the detection reference 40 at a predetermined initial position,e.g., at the position taken by the midair image 30 or at a positionslightly above the position taken by the midair image 30, before theoperation proceeds to step S24. At this time, a message “Calibrationexecuted during icon operation” is briefly displayed in the midair image30. However, it is not essential that this message be displayed.

As the user moves his finger downward in order to perform an operationat the display position of an icon 30A, the operation detector 13 startsdetection of finger movement in step S24, and then the operationproceeds to step S25. In step S25, the detection reference control unit204 makes a decision based upon the detection output provided by theoperation detector 13 as to whether or not the finger moving downwardhas passed through the detection reference 40. If an affirmativedecision is made in step S25, i.e., if the finger moving downward haspassed through the detection reference 40 and has moved further down,the operation proceeds to step S26. F1 in FIG. 11 indicates the fingerhaving passed through the detection reference 40 during its downwardmovement. In step S26, the detection reference control unit 204, havingdecided that the finger F1 has reached the detection reference 40, i.e.,it has passed through the detection reference 40, executes icon displayswitchover so as to switch the midair image 30 in correspondence to theicon 30A having been operated. The operation then proceeds to step S27.In step S27, the detection reference control unit 204 makes a decisionas to whether or not the finger F1 has reached the reach position 50,and if an affirmative decision is made, the operation proceeds to stepS28, whereas if a negative decision is made, the operation is held untilan affirmative decision is made. In step S28, the detection referencecontrol unit 204 adjusts the display mode for the icons 30A based uponthe distance between the reach position 50 and the detection reference40, and then the operation proceeds to step S33. In more specific terms,the image generation unit 201 alters the display positions along adirection in which the icons 30A move away from the vanishing point andincreases the size of the icons 30A, as shown in FIG. 9(c).

It is to be noted that if the detection reference control unit 204detects in step S28 that the reach position 50 is not located furtherdownward (toward the Z direction − side) beyond a predetermined rangerelative to the detection reference 40 by comparing the reach position50 and the position of the detection reference 40, the image generationunit 201 does not need to adjust the display mode for the icons 30A. Asan alternative, the image generation unit 201 may set the extent towhich the display mode for the icons 30A is to be altered (i.e., thedisplacement quantity for the icons 30A) to 0 (in other words, thedisplay mode for the icons 30A may remain substantially unchanged).

If a negative decision is made in step S25, i.e., if the finger movingdownward has not passed through the detection reference 40, theoperation proceeds to step S29. In step S29, the detection referencecontrol unit 204 makes a decision based upon the detection outputprovided by the operation detector 13 as to whether or not the fingerhas reached the reach position 50, and if an affirmative decision ismade, the operation proceeds to step S30. If, on the other hand, anegative decision is made, the operation is held until an affirmativedecision is made. A finger F2 in FIG. 11 indicates that the reachposition 50 is in alignment with the detection reference 40. In stepS30, the detection reference control unit 204 makes a decision basedupon the detection output provided by the operation detector 13 as towhether or not the reach position 50 is in alignment with the detectionreference 40, and if an affirmative decision is made, the operationproceeds to step S31, whereas if a negative decision is made, theoperation proceeds to step S32. In step S31, icon display switchover isexecuted since the reach position 50 is in alignment with the detectionreference 40 without adjusting the display mode for the icons 30A,before the operation proceeds to step S33.

It is to be noted that the detection reference control unit 204 maydecide that the reach position 50 is at the detection reference 40 evenwhen the reach position 50 is not exactly in alignment with thedetection reference 40, e.g., when the reach position 50 is detectedwithin a predetermined range from the detection reference 40. In such acase, the detection reference control unit 204 may make a decision instep S25 in FIG. 12 as to whether or not the reach position 50 islocated further downward relative to the predetermined range from thedetection reference 40. Then, if the reach position 50 is locatedfurther downward beyond the predetermined range from the detectionreference 40, an affirmative decision will be made in step S25 and theoperation will proceed to step S26.

In addition, the detection reference control unit 204 may make adecision in step S30 as to whether or not the reach position 50 islocated within a predetermined range relative to the detection reference40. In this case, if the furthest reach position 50 is located withinthe predetermined range from the detection reference 40, an affirmativedecision will be made in step S30 and the operation will proceed to stepS31.

In step S32, with the reach position 50 located above the detectionreference 40 as indicated by a finger F3 in FIG. 11, the detectionreference control unit 204 adjusts the display mode for the icons 30Abased upon the distance between the reach position 50 and the detectionreference 40. More specifically, the image generation unit 201 altersthe display positions of the icons 30A along a direction toward thevanishing point and reduces the size of the icons 30A, as shown in FIG.9(b). In step S33, a decision is made as to whether or not an endoperation has been performed in order to exit the second calibrationprocessing mode, and if an affirmative decision is made, the secondcalibration processing mode is terminated, whereas if a negativedecision is made, the operation returns to step S24.

In the second calibration processing mode described above, which isexecuted concurrently while the midair image operation mode is underway,the user is able to perform an operation at the display position of themidair image 30 by using the detection reference 40 optimized for theuser without having to be aware that calibration processing is also inprogress. It is to be noted that the first/second calibration processingmode selection does not necessarily need to be made by the user andinstead, the display device 1 may automatically select either the firstcalibration processing mode or the second calibration processing mode.In addition, it is not essential that both the first calibrationprocessing mode and the second calibration processing mode may beavailable and only one of these calibration modes may be available.

In the first embodiment, the image generation unit 201 adjusts thedisplay mode for an icon 30A constituting a first image based upon anoperation detected by the operation detector 13. More specifically, theimage generation unit 201 adjusts the display mode for the icon 30Arelative to the reference lines 310 through 314 used as a depth cuebased upon the positional relationship between the detection reference40 and the position at which the user operation is detected, obtainedvia the operation detector 13. As a result, a user perception can becreated as if the spatial position of the icon 30A has changed, so as tolead the user to adjust the position at which he performs a subsequentoperation with respect to the display position of the midair image 30.

In addition, the image generation unit 201 in the first embodimentadjusts the display mode for the icon 30A so as to create a userperception as if the display position of the icon 30A has moved towardor further away from the user. As a result, a perception can be createdin the user as if the spatial position of the icon 30A along the Zdirection has changed so as to lead the user to adjust the positionalong the Z direction at which he performs an operation in relation tothe display position of the midair image 30. In other words, detectionof a user operation at the detection reference 40 is enabled.

Moreover, the image generation unit 201 in the first embodimentdetermines the direction along which the display position of the icon30A is to be altered based upon the positional relationship between thedetection reference 40 and the position at which the user operation hasbeen detected. This means that the direction along which the icon 30A isto be perceived to move in space is determined based upon the useroperation performed during the calibration processing and thus, it isensured that a subsequent user operation is performed at the displayposition at which the midair image 30 is displayed.

In addition, the image generation unit 201 in the first embodimentalters the display position of the icon 30A along a direction (towardthe Z direction − side) determined based upon a press-down operationperformed by the user on the midair image 30 or along the oppositedirection (toward the Z direction+ side). With the user led to adjustthe position at which he performs an operation along the Z direction inthis manner, detection of a user operation at the detection reference 40is ensured.

Moreover, the image generation unit 201 in the first embodiment adjuststhe display of the icon 30A so that the display position of icon 30Adisplayed in space, is altered along the direction determined based upona press-down operation performed by the user when the press-downoperation performed by the user does not reach the detection reference40 (see FIG. 7(b)). Namely, a perception is created in the user as ifthe position of the icon 30A has been adjusted to move further away fromthe user. Since this leads the user to perform a press-down operation ata position further downward, it is ensured that a press-down operationby the user reaches the detection reference 40.

Furthermore, the image generation unit 201 in the first embodimentadjusts the display of the icon 30A so that the display position of theicon 30A displayed in space is altered along the direction opposite fromthe direction determined based upon a press-down operation performed bythe user when the user press-down operation has reached the detectionreference (see FIG. 7(d)). Namely, a perception is created in the useras if the position of the icon 30A has been adjusted to move closer tothe user. Since this leads the user to perform a press-down operation ata position closer to the user, it is ensured that the user performs apress-down operation at the detection reference 40.

In addition, in the first embodiment, the position at which a useroperation is detected is the reach position 50. Since this allows thedisplay of the icons 30A to be adjusted based upon the position at whichthe user perceives that he has “touched” the display position of themidair image 30, the accuracy of the calibration processing can beimproved.

The image generation unit 201 in the first embodiment adjusts thedisplay of the icon 30A when a user operation is not detected at thedetection reference 40. Since this leads the user to perform apress-down operation at a position further down, it is ensured that thepress-down operation performed by the user is detected at the detectionreference 40.

In addition, the image generation unit 201 in the first embodiment makesan adjustment so as to reduce the size of the icon 30A when a useroperation is not detected at the detection reference 40. Since thiscreates a user perception as if the display of the icon 30A in space hasmoved further away from the user, the user is led to perform anoperation at a position that aligns with position of the detectionreference 40.

In addition, when the position at which the user operation is detectedis closer to the user relative to the detection reference 40, the imagegeneration unit 201 in the first embodiment adjusts the display of theicon 30A relative to the reference lines 310 through 314 so as to createa user perception as if the position of the icon 30A has moved furtheraway from the user. Thus, the user, perceiving as if the midair displayof the icon 30A has moved further away from the user, is led to performa user operation at a position that aligns with the position ofdetection reference 40.

In addition, when the position at which a user operation is detected setapart from the detection reference 40 over a second distance greaterthan a first distance and is located further toward the user, the imagegeneration unit 201 in the first embodiment adjusts the display of theicon 30A as described below. Namely, the image generation unit 201adjusts the display of the icon 30A relative to the reference lines 310through 314 so as to create a user perception as if the display positionof the icon 30A has moved further away from the user by an extentgreater than the extent to which the display position moves when theuser operation detection position is set apart over the first distance.This means that the image generation unit 201 adjusts the display of theicon 30A relative to the reference lines 310 through 314 so as to createa user perception as if the display position of the icon 30A movesfurther away from the user by a greater extent as the distance betweenthe user operation detection position and the detection reference 40increases. Through these measures, it is ensured that a user operationreaches the detection reference 40 by creating different userperceptions with respect to the depth of the icon 30A based upon thedistance between the detected user operation and the detection reference40.

Moreover, the image generation unit 201 in the first embodiment adjuststhe display mode for the icon 30A so as to create a user perception asif the display position of the icon 30A has changed along the opticalaxis of the image-forming optical system 12. This means that a useroperation can be detected at the detection reference 40 by leading theuser to adjust the user operation position along the optical axis. It isto be noted that the present invention is not limited to an example inwhich a user perception is created that the display position of icon 30Ahas changed precisely along the optical axis of the image-formingoptical system 12. In other words, the image generation unit 201 mayadjust the display mode for the icon 30A so as to create a userperception as if the display position of the icon 30A has moved along adirection at an angle relative to the Z axis, i.e., along a directionother than the direction perpendicular to the optical axis of theimage-forming optical system 12.

(Variation 1 of the First Embodiment)

In the first embodiment described above, the display mode for an icon30A or 300A is adjusted by altering its display position and size. Otherexamples that may be adjusted in order to adjust the display mode forthe icon 30A or 300A are described as variation 1 below.

Example 1

The image generation unit 201 adjusts the display mode by altering thesharpness with which an icon 30A in FIG. 3(a) or the icon 300A in FIG.4(a) is displayed. The image generation unit 201 increases the sharpnessof the icon 30A or 300A so as to create a user perception as if the icon30A or 300A has moved closer to the user. The image generation unit 201decreases the sharpness of the icon 30A or 300A so as to create a userperception as if the icon 30A or 300A has moved further away from theuser. In this case, too, the reference lines 310 through 314 are used asa depth cue, as in the first embodiment. It is to be noted that if thesharpness of only one icon 30A among the plurality of icons 30A shown inFIG. 3(a) is altered, the other icons 30A, the sharpness of whichremains unchanged, are used as a depth cue. The image generation unit201 adjusts the sharpness by altering the degree of blur with which theicon 30A or 300A is displayed or by altering the degree of whiteness(haziness) in the display of the icon 30A or 300A based upon the conceptof air perspective (the laws of air perspective).

The image generation unit 201 creates a user perception as if the icon30A or 300A has moved closer to the user by reducing the degree of blurat the edges of the icon 30A or the icon 300A, or at a character, apattern or the like superimposed over the icon and thus displaying itwith better definition. The image generation unit 201 creates a userperception as if the icon 30A or 300A has moved further away from theuser by increasing the degree of blur at the edges of the icon 30A or300A or at a character, a pattern or the like superimposed thereupon. Inaddition, the image generation unit 201 creates a user perception ad ifthe icon 30A or 300A has moved closer to the user by decreasing thedegree of whiteness rendered at the icon 30A or 300A. The imagegeneration unit 201 creates a user perception as if the icon 30A or 300Ahas moved away the user by increasing the degree of whiteness renderedat the icon 30A or 300A. It is to be noted that the image generationunit 201 may alter only either the degree of blur or the degree ofwhiteness of the icon 30A or the icon 300A or it may alter both thedegree of blur and the degree of whiteness of the icon 30A or the icon300A.

It is to be noted that in the example described above, the display modeis adjusted by altering the display sharpness of icon 30A or 300A.However, the image generation unit 201 may alter the display position ofthe icon 30A or 300A and its size or may alter either the displayposition or the icon size, in addition to altering the sharpness of theicon 30A or 300A.

In example 1 of variation 1, the image generation unit 201 alters thesharpness with which the icon 30A or 300A is displayed based upon thepositional relationship between the user operation and the detectionreference 40. As a result, the user, perceiving as if the displayposition at which the icon 30A or 300A is displayed in space has moved,is led to adjust the position at which he performs an operation withrespect to the midair image 30.

Example 2

The image generation unit 201 adjusts the display mode by altering thecolor with which an icon 30A in FIG. 3(a) or the icon 300A in FIG. 4(a)is displayed. Under normal circumstances, an object taking on a coldcolor is perceived to be located further away than an object in the samesize taking on a warm color. The image generation unit 201 creates auser perception as if the icon 30A or 300A has moved further away fromthe user by rendering the icon 30A or 300A in a cold, receding color,such as blue. The image generation unit 201 creates a user perception asif the icon 30A or 300A has moved closer to the user by rendering theicon 30A or 300A in a warm, advancing color, such as red or yellow. Inthis case, the color of the reference lines 310 through 314 is used as adepth cue, as in the first embodiment. It is to be noted that if thecolor of only one icon 30A among the plurality of icons 30A shown inFIG. 3(a) is altered, the other icons 30A, the color of which remainsunchanged, are used as a depth cue.

It is to be noted that in the example described above, the display modeis adjusted by altering the color of the icon 30A or 300A. However, theimage generation unit 201 may alter the display position of the icon 30Aor 300A and its size or may alter either the display position or theicon size, in addition to altering the color of icon 30A or 300A.

Example 3

The image generation unit 201 adjusts the display mode by altering theluminance with which an icon 30A in FIG. 3(a) or the icon 300A in FIG.4(a) is displayed. The image generation unit 201 decreases the luminanceof the icon 30A or 300A, thus rendering it darker, so as to create auser perception as if the icon 30A or 300A has moved further away fromuser. The image generation unit 201 increases the luminance of the icon30A or 300A and thus renders it brighter so as to create a userperception as if the icon 30A or 300A has moved closer to the user. Inthis case, too, the luminance of the reference lines 310 through 314 isused as a depth cue, as in the first embodiment. It is to be noted thatif the luminance of only one icon 30A among the plurality of icons 30Ashown in FIG. 3(a) is altered, the other icons 30A, the luminance ofwhich remains unchanged, are used as a depth cue.

In this example, as the luminance of the icon 30A or 300A is raised andthus its brightness increases, the user may perceive that the icon 30Aor 300A is located closer to the user. In addition, the following effectmay be created. Namely, as the icon 30A or the icon 300A is renderedbrighter, the user is able to view the icon 30A or 300A with betterclarity and thus the ease of operation improves.

It is to be noted that in the example described above, the display modeis adjusted by altering the luminance of the icon 30A or 300A. However,the image generation unit 201 may alter the display position of the icon30A or 300A and their size or may alter either the display position orthe icon size, in addition to altering the luminance of the icon 30A or300A.

Example 4

The image generation unit 201 adjusts the display mode by altering theextent to which the icon 30A or 300A overlaps with the reference lines310 through 314 used as a depth cue. FIG. 13 provides schematicillustrations, each showing an example of a degree to which an icon 30Amay overlap with the reference lines 310 through 314. In FIG. 13(a), thereference line 310 is partially obscured by the icon 30A, whereas inFIG. 13(b), the icon 30A is partially obscured by the reference line310. It is to be noted that the icon 30A in FIG. 13(a) and the icon 30Ain FIG. 13(b) are displayed in the same size and at the same position.When the reference line 310 is partially obscured behind the icon 30A,as shown in FIG. 13(a), a perception is created that the icon 30A islocated closer than the icon 30A part of which ranges outside thereference line 310 and is obscured, as shown in FIG. 13(b). The imagegeneration unit 201 uses this effect to create a user perception as ifthe icon 30A or 300A has moved further away from the user, by displayingthe icon 30A or 300A so that it appears to be partially obscured behinda reference line. The image generation unit 201 creates a userperception as if the icon 30A or 300A has moved closer to the user, bydisplaying the icon 30A or 300A so that the reference line appears to bepartially obscured behind the icon 30A or 300A. In this case, the imagegeneration unit 201 creates a user perception as if the icon 30A or 300Ahas moved along the Z direction without actually moving the displayposition of the icon 30A or 300A on the midair image 30.

It is to be noted that the icon 30A or 300A is displayed at the sameposition in FIG. 13(a) and FIG. 13(b). The image generation unit 201 mayalso alter the display position and the size of the icon 30A or 300A oralter either the display position or the size of the icon 30A or 300A inaddition to altering the degree of overlap with respect to the icon 30Aor 300A.

In example 4 of variation 1, the image generation unit 201 alters thedegree to which the icon 30A or 300A overlaps a reference line basedupon the positional relationship between the user operation and thedetection reference 40. As a result, the user, perceiving as if thedisplay position of the icon 30A or 300A in space has moved, is led toadjust the position at which he performs an operation with respect tothe midair image 30.

In addition, if detection references 40, each set in correspondence toone of three icons 30A, form steps along the Z direction as shown inFIG. 3(c), the image generation unit 201 may display the individualicons 30A in the midair image 30 with overlaps, as shown in FIG. 13 (c).Namely, the icon 30A corresponding to the detection reference 40 set atthe position closest to the midair image 30 (located toward the leftside of the drawing sheet (toward the X direction − side)), among theicons 30A in FIG. 3(c), is displayed so that it is partially obscured bythe icon 30A located at the center along the X direction among the threeicons 30A. In addition, the icon 30A at the center along the X directionamong the three icons 30A, is displayed so that it is partially obscuredby the icon 30A corresponding to the detection reference 40 set at theposition furthest from the midair image 30 (located toward the rightside of the drawing sheet (toward the X direction+ side)), among thethree icons 30A. It is to be noted that while FIG. 13(c) shows theindividual icons 30A taking different positions along the Y direction tosimplify the illustration, they may be displayed at positions inalignment with one another along the Y direction. The image generationunit 201 may set the extent to which the individual icons 30A areobscured, i.e., the extent of overlap, based upon the distances betweenthe midair image 30 and the detection references 40.

As a result, the user experiences a perception as if the individualicons 30A are displayed at different positions from one another alongthe Z direction. This, in turn, makes it possible to detect the useroperation at the corresponding detection reference 40 even when thedetection references 40 corresponding to the individual icons 30A areset apart from the operation detector 13 over distances H2 differentfrom one another.

It is to be noted that the image generation unit 201 may incorporate aneffect rendered through motion parallax. Namely, the image generationunit 201 may adjust the display mode for the icons 30A as shown in FIG.13(c), by moving one of the icon 30A and the other icons 30A, from theinitial display state shown in FIG. 3(a) and altering the extent oftheir overlap as time elapses. Through these measures, the perceivedmovement of the icons 30A along the depthwise direction can beaccentuated.

Example 5

The image generation unit 201 adjusts the display mode by altering ashadow added to the icon 30A or 300A. FIG. 14 provides schematicillustrations, each showing an icon 300A and an example of a shadowimage 315 that may be formed when hypothetical light is cast on the icon300A from above. Namely, the shadow image 315 is used as a depth cue forthe user in perceiving the position of the icon 300A along the Zdirection in example 5. The areas of the shadow image 315 formed incorrespondence to the icon 300A are different from one another in FIG.14(a), FIG. 14(b) and FIG. 14(c). It is to be noted that the icon 300Aassumes the same size and the same display position in the midair image300 in FIG. 14(a) through FIG. 14(c).

The shadow image 315 in FIG. 14(b) assumes an area smaller than that ofthe shadow image 315 in FIG. 14(a). The icon 300A in FIG. 14(b), withthe shadow image 315 assuming a smaller area, is perceived to be setapart from the shadow image 315 by a smaller distance compared to thedistance between the icon 300A shown in FIG. 14 (a) and itscorresponding shadow image 315. Namely, the user perceives as if theicon 300A shown in FIG. 14(b) is located further away from the usercompared to the icon 300A shown in FIG. 14 (a).

The icon 300A shown in FIG. 14(c), with the corresponding shadow image315 assuming a large area, will be perceived to be set apart from theshadow image 315 over a greater distance compared to the distancebetween the icon 300A shown in FIG. 14(a) and its shadow image 315.Namely, the user perceives as if the icon 300A shown in FIG. 14(c) islocated closer to the user compared to the icon 300A shown in FIG.14(a). The image generation unit 201 incorporates this effect andcreates a user perception as if the icon 30A or 300A has moved furtheraway from or closer to the user by controlling the area of the shadowimage 315 corresponding to the icon 30A or 300A. Assuming that themidair image 300 in FIG. 14(a) is brought up in the initial display, theimage generation unit 201 brings up on display the midair image 300 inFIG. 14(b) if the reach position 50 is located above the detectionreference 40 (see FIG. 7(b)) during the calibration processing. If thereach position 50 is below the detection reference 40 (see FIG. 7(d))during the calibration processing, the image generation unit 201 bringsup on display the midair image 300 in FIG. 14(c). In other words, whenthe reach position 50 is not detected at the detection reference 40, theimage generation unit 201 displays the shadow image 315, as shown inFIG. 14(b) or FIG. 14(c) based upon the positional relationship betweenthe reach position 50 and the detection reference 40.

Instead of adjusting the size of the shadow image 315 that appears to beformed by the icon 30A or 300A as described above, the image generationunit 201 may alter an image of a shade formed at an icon 300A. FIG. 15provides schematic illustrations, each showing a round icon 300A andpresenting an example of a shade image added into the icon 300A as ameans for display mode adjustment. FIG. 15(a), FIG. 15(b) and FIG. 15(c)are schematic illustrations, respectively presenting an example of theicon 300A that may be brought up in the initial display, an example of ashade image 315 formed over an upper area (toward the Y direction −side) at the round icon 300A and an example of a shade image 315 formedover the lower area (toward the Y direction+ side) at the icon 300A.

The image generation unit 201 forms a shade image 315 at the icon 300Ashown in FIG. 15(b), by graduating the luminance of the shade image 315from lowest at the top (the Y direction − side) to highest at the bottom(the Y direction+ side). With such a shade image 315 formed therein, theicon 300A in FIG. 15(b) can be perceived to recede toward the sideopposite from the user (toward the Z direction − side) in comparison tothe icon 300A in the initial display shown in FIG. 15(a). Namely, theicon 300A in FIG. 15(b) can be perceived to be located further away fromthe user compared to the icon 300A in the initial display. The icon 300Ashown in FIG. 15(c), includes a shade image 315 formed by the imagegeneration unit 201 by graduating the luminance of the shadow image 315from lowest at the bottom (the Y direction+ side) to highest at the top(the Y direction − side). With such a shade image 315 formed therein,the icon 300A in FIG. 15(c) can be perceived to advance toward the user(toward the Z direction+ side) in comparison to the icon 300A in theinitial display shown in FIG. 15(a). Namely, the icon 300A in FIG. 15(c)can be perceived to be located closer to the user compared to the icon300A in the initial display. Using this effect, the image generationunit 201 creates a user perception as if the icon 30A or 300A has movedfurther away from the user by forming the shade image 315 over an upperarea in the icon 30A or 300A. Furthermore, the image generation unit 201creates a user perception as if the icon 30A or 300A has moved closer tothe user by forming the shade image 315 over a lower area in the icon30A or 300A.

It is to be noted that the shade images 315 that may be formed at theicon 30A or 300A by the image generation unit 201 are not limited tothose shown in FIG. 15. The image generation unit 201 may form a shadeimage 315 in the icon 30A or 300A by using, for instance, the“Ramachandran” crater illusion.

In example 5 of variation 1, the image generation unit 201 executesprocessing for adding a shadow/shade image at the icon 30A or 300A basedupon the positional relationship between the user operation and thedetection reference 40. As a result, the user perceiving as if thedisplay position of the icon 30A or 300A has moved, is led to adjust theuser operation position.

The image generation unit 201 may adjust the display mode by adoptingone of the display mode adjustment methods in the first embodiment andexamples 1 through 5, or it may execute display mode adjustment bycombining a plurality of display mode adjustment methods. Namely, theimage generation unit 201 may adjust at least one of; the displayposition of the icon 30A, the size of the icon 30A, the sharpness of theicon 30A and a shadow/shade added to the icon 30A, based upon thepositional relationship between the user operation position and thedetection reference 40. In this situation, the image generation unit 201may alter at least one of; the display position, the size, the sharpnessand the shadow/shade, with respect to the icon 30A, without displayingthe reference lines 310 through 314. In addition, based upon thepositional relationship between the user operation position and thedetection reference 40, the image generation unit 201 may alter at leastone of; the display position, the size and the luminance of the icon30A, among the plurality of display mode adjustment examples explainedabove.

(Variation 2 of the First Embodiment)

The display device 1 in variation 2, uses a physical object as a depthcue for the second reference and displays an icon 30A or 300A in amidair image 30, without displaying the reference lines 310 through 314.

The display device 1 in variation 2 may adopt a structure similar tothat of the display device 1 in the first embodiment illustrated in FIG.1 and FIG. 2.

FIG. 16 presents an external view of the display device 1 in variation 2in a perspective. At a surface of a body 10, (a surface located towardthe Z direction+ side, hereafter to be referred to as a frame member101), index marks 102 to be used as a guide that will aid the userlooking at a midair image 30 to ascertain the display position of themidair image 30 in space are disposed. The index marks 102 may beengraved in the frame member 101 or they may be decals or the likeapplied on the frame member 101. In other words, the index marks 102used as the second reference are physical objects. In the examplepresented in FIG. 16, the index marks 102 are rectangular marks. It isto be noted that the index marks 102 may be round marks instead ofrectangular marks, they may be symbols such as arrows or stars, or theymay be characters such as alphabet letters. In addition, while one indexmark 102 is disposed on each of the four sides of the frame member 101in the example presented in FIG. 16, the present invention is notlimited to this example and a plurality of index marks 102 may bedisposed on each side of the frame member 101. Furthermore, index marks102 do not need to be disposed on all sides of the frame member 101, andan index mark 102 may be disposed on at least one side.

FIG. 17 shows icons 30A in a midair image 30 displayed at the displaydevice 1 in variation 2 and index marks 102 disposed at the frame member101. FIG. 17(a) shows the icons 30A in the initial display brought upbefore calibration processing is executed. FIGS. 17(b) and 17(c) eachshow the icons 30A displayed in a display mode adjusted by the imagegeneration unit 201 through calibration processing. In the examplepresented in FIG. 17(b), the display mode for the icons 30A has beenadjusted by reducing the size of the icons 30A in comparison to the sizeof the icons 30A in FIG. 17(a). In the example presented in FIG. 17(c),the display mode for the icons 30A has been adjusted by increasing thesize of the icons 30A in comparison to the size of the icons 30A in FIG.17(a).

Prior to the calibration processing, the user, looking at the icons 30Abrought up in the initial display, as shown in FIG. 17 and the indexmarks 102, ascertains the positions taken by the icons 30A in space(positions taken along the Z direction) in reference to the index marks102. The display of the icons 30A in FIG. 17(a) may be adjusted to thatshown in FIG. 17(b) as a result of the calibration processing. In thiscase, the size of the icons 30A on display is reduced in comparison tothat of the icons 30A in FIG. 17(a). Thus, the user, looking at theicons 30A by using the index marks 102 for reference, experiences aperception as if the icons 30A have moved further away from the userrelative to those in FIG. 17(a). In addition, the display of the icons30A in FIG. 17(a) may be adjusted to that shown in FIG. 17(c) as aresult of the calibration processing. In this case, the size of theicons 30A on display is increased in comparison to that of the icons 30Ain FIG. 17(a). Thus, the user, looking at the icons 30A by using theindex marks 102 for reference, experiences a perception as if the icons30A have moved closer to the user relative to those in FIG. 17(a).

It is to be noted that instead of providing the index marks 102 at theframe member 101, an index mark 102 may be displayed at a display unitsuch as a liquid crystal monitor. In such a configuration, a displayunit 103 will be disposed over the upper surface of the frame member 101of the display device 1, as shown in FIG. 18(a). FIG. 18(b) is a blockdiagram that includes the control unit 20, as well as the display unit11, the operation detector 13, the display unit 103, which arecontrolled by the control unit 20, among the structural elementsconstituting this display device 1. The control unit 20 further includesan index display control unit 209 that controls display at the displayunit 103, in addition to the functions of the control unit 20 in thefirst embodiment shown in FIG. 2. The index display control unit 209controls the display unit 103 to start displaying an index mark 102 whenthe display device 1 is started up to begin operation in the midairimage operation mode or when the calibration processing mode is startedup at a startup of display device 1. In other words, the index displaycontrol unit 209 starts display at the display unit 103 in step S2 inthe flowchart presented in FIG. 10 or step S22 in the flowchartpresented in FIG. 12, in reference to which the first embodiment hasbeen explained. The index display control unit 209 brings up on displayat the display unit 103 any of the various types of symbols, charactersor the like mentioned earlier as an index mark 102. As a result, theuser is able to perceive apparent movement of the icons 30A or 300Aalong the depthwise direction in reference to the index mark 102displayed at the display unit 103, in the same way as when an index mark102 disposed at the frame member 101 is used.

It is to be noted that while the image generation unit 201 adjusts thedisplay mode by altering the size of the icons 30A or 300A in variation2 described above, the present invention is not limited to this example.The image generation unit 201 may adjust the display mode by alteringthe sharpness, the color or the luminance, as has been explained inreference to the various examples of variation 1.

(Variation 3 of the First Embodiment)

The display device 1 in the first embodiment or variation 1 or 2 thereofdescribed above designates the lowest point to which the user'sfingertip reaches as it first moves downward in order to perform anoperation at an icon display position and then moves upward over aspecific distance, as the reach position. The display device 1 thenadjusts the display mode for the midair image based upon the distancebetween the reach position and the detection reference. The displaydevice 1 in variation 3 instead calculates the velocity or theacceleration of the user's fingertip F based upon the detection outputprovided by the operation detector 13, and predicts a reach position ofthe user's fingertip F based upon the velocity or the accelerationhaving been calculated. The display device 1 then adjusts the displaymode for the midair image based upon the distance between the predictedreach position and the detection reference. FIG. 19 is a block diagramshowing the control unit 20 as well as the display unit 11 and theoperation detector 13 controlled by the control unit 20, among thestructural components in the display device 1 in variation 3.

The display device 1 in variation 3 will be described by focusing on itsstructural features different from those in the display device in thefirst embodiment. A velocity·acceleration detection unit 206 in FIG. 19reads out the electrostatic capacitance value detected by the operationdetector 13 after predetermined time intervals, calculates the velocityof the finger movement based upon a change occurring in theelectrostatic capacitance value over each predetermined time intervaland also calculates the acceleration of the finger movement based uponthe velocity having been calculated. A reach position predicting unit207 predicts the reach position for the finger based upon the fingermovement velocity or acceleration output by the velocity·accelerationdetection unit 206. The reach position predicting unit 207 is able toestimate the reach position for the finger by, for instance, detectingthat the moving finger, having been accelerating or moving at asubstantially constant speed, has shifted into a decelerating state andascertaining the rate of deceleration. The image generation unit 201adjusts the display mode for the midair image 30 based upon the reachposition predicted by the reach position predicting unit 207.

Next, the processing executed in the first calibration processing modein the display device 1 in variation 3 will be explained in reference toFIG. 20 and FIG. 21. The processing executed in steps other than stepS104 through step S107 in the flowchart presented in FIG. 21 isidentical to that in the flowchart presented in FIG. 10, andaccordingly, a repeated explanation is not provided. As the fingertip Fmoves into the predetermined detection range 13A of the operationdetector 13, as shown in FIG. 11(a), the operation detector 13 detectsthe movement of the fingertip F as a change in the electrostaticcapacitance value in step S104. In step S105, the velocity·accelerationdetection unit 206 calculates the velocity or the acceleration of themovement of the fingertip F based upon the detection output provided bythe operation detector 13. In step S106, the reach position predictingunit 207 calculates the reach position for the fingertip F based uponthe velocity or the acceleration of the movement having been calculatedby the velocity·acceleration detection unit 206. The reach position forthe finger calculated by the reach position predicting unit 207, i.e.,predicted by the reach predicting unit 207, is indicated by a dottedline 60 in FIG. 20(b). In step S107, the image generation unit 201adjusts the display mode for the icon 300A based upon the predictedreach position 60, as indicated in FIG. 20(b). The image generation unit201 stores data indicating the change quantity indicating the extent towhich the size of the icon 300A is altered and the displacementquantity, with respect to the extent to which the display position ismoved, into the storage unit 205. In a subsequent step S110, the imagegeneration unit 201 generates, based upon the stored data, display imagedata expressing a midair image 30 with the display mode for an icon 30Aadjusted, and the display control unit 202 brings up the display data ondisplay as a midair image 30 in the midair image operation mode. It isto be noted that the reach position for the finger may be predictedbased upon both the velocity and the acceleration of the finger movementor based upon either one of them.

It is to be noted that the velocity·acceleration detection unit 206reads out the electrostatic capacitance value detected by the operationdetector 13 after predetermined time intervals, calculates the velocityof the finger movement based upon a change occurring in theelectrostatic capacitance value over each predetermined time intervaland calculates the acceleration of the finger movement based upon thevelocity thus calculated in the description provided above. However, thepresent invention is not limited to this method and it may be adopted inconjunction with a velocity·acceleration detection unit 206 configuredwith an image-capturing device. In addition, while thevelocity·acceleration detection unit 206 calculates the velocity or theacceleration of the user's finger movement in the example describedabove, the velocity or the acceleration of the movement of the user'sfoot or elbow or the movement of a stylus pen held by the user may becalculated instead.

It is to be also noted that the reach position predicting unit 207calculates a predicted reach position 60 for the user's finger basedupon the velocity or the acceleration of the movement of the user'sfinger having been calculated and the image generation unit 201 adjuststhe display mode for the midair image 30 based upon the predicted reachposition 60 calculated for the user's finger. However, the reachposition predicting unit 207 does not need to determine the predictedreach position 60 for the user's finger for each operation. If apredicted reach position 60 is calculated based upon an unintendedmovement of the user's finger occurring prior to a user operation, thedisplay mode for the icon 30A or 300A may be adjusted too frequently andit may become difficult to guide the user's fingertip F to the optimalposition. Such an undesirable result can be prevented by engaging thereach position predicting unit 207 in calculation of a reach position 60and the image generation unit 201 in adjustment of the display mode forthe midair image 30 based upon the predicted reach position 60 only whenthe velocity·acceleration detection unit 206 has detected a velocity andan acceleration of the user's finger movement each represented by avalue equal to or greater than a predetermined threshold value.

In variation 3, in which the reach position 50 for the finger ispredicted based upon the velocity or the acceleration of fingermovement, calibration processing can be executed promptly.

While the calibration processing in this variation is adopted in thefirst calibration processing mode in the first embodiment in the exampledescribed above, the calibration processing may also be adopted in thesecond calibration processing mode. In the latter case, the processingin step S105 and step S106 in the flowchart presented in FIG. 21 will beexecuted after step S24 in the flowchart presented in FIG. 12 inreference to which the first embodiment has been described.Subsequently, the display mode for the midair image 30 will be adjustedbased upon the predicted reach position 60 calculated in step S28 andstep S32 without executing the processing in step S27 and step S29 inFIG. 12. By adopting variation 3 in the second calibration processingmode, it becomes possible to estimate in advance the reach position forthe fingertip F of the user performing a midair image operation beforethe fingertip F reaches the detection reference 40 and the display modefor the midair image 30 can be adjusted based upon the predicted reachposition. This means that even when the fingertip F of the user does notreach the detection reference 40, the user is comfortably able toperform an operation since an error such as a failure to execute icondisplay switchover can be prevented.

(Variation 4 of the First Embodiment)

In the first embodiment and variations 1 through 3 of the firstembodiment, the display device 1 detects or predicts the reach position,and the image generation unit 201 adjusts the display mode for themidair image 30 based upon the distance between the reach position 50and the detection reference 40 through a single session of calibrationprocessing. As an alternative, the image generation unit 201 adjusts thedisplay mode for the midair image 30 in the midair image operation modebased upon the distances between the reach positions detected through aplurality of sessions of calibration processing and the detectionreference in the display device 1 in variation 4.

In the first session of the calibration processing, the detectionreference control unit 204 determines the reach position 50 for thefinger based upon the detection output provided from the operationdetector 13. The image generation unit 201 calculates a change quantitywith respect to the size of the icon 300A and a displacement quantitywith respect to the display position based upon the reach position 50.The image generation unit 201 stores data indicating the change quantityand the displacement quantity thus calculated into the storage unit 205.A second session of the calibration processing is executed in successionand data indicating a change quantity and a displacement quantity arestored into the storage unit 205 in a similar manner. A third session ofthe calibration processing may be executed in succession following thissession. Based upon the data indicating a plurality of sets of changequantities and displacement quantities stored in the storage unit 205through the plurality of sessions of calibration processing having beenexecuted successively as described above, the image generation unit 201selects a single set of change quantity and displacement quantity for anicon 30A in a midair image 30 displayed in the midair image operationmode.

The display mode for a given icon 30A may be determined by using theplurality of sets of change quantities and displacement quantitiesthrough any of various procedures. For instance, the image generationunit 201 may calculate a single change quantity or displacement quantityfor the icon 30A as the arithmetic mean of the plurality of changequantities or displacement quantities or as the geometric mean of theplurality of sets of change quantities and displacement quantities. Asan alternative, the image generation unit 201 may determine a new changequantity or displacement quantity by applying optimal weight to each ofthe plurality of change quantities or displacement quantities. Forinstance, the image generation unit 201 may calculate a change quantityor displacement quantity for the icon 30A by weighting the changequantity or the displacement quantity H_(N) determined through an Nthsession and the change quantity or displacement quantity H_(N)+1determined through an N+1th session at a ratio of 3:7. In more specificterms, using H_(N) and H_(N)+1, the image generation unit 201 calculatesa change quantity or displacement quantity for the icon 30A based uponthe results of calculation executed as expressed as;(H_(N)×3+H_(N)+1×7)/10. The weighting ratio used in this calculation isnot limited to 3:7 and the number of sessions is not limited to 2,either. Furthermore, it will be obvious that instead of individuallycalculating a change quantity and a displacement quantity based upon thereach position for the finger and storing them into the storage unit 205in correspondence to each of the plurality of sessions of thecalibration processing, reach positions for the finger, each detected incorrespondence to a plurality of sessions of the calibration processing,may be stored into the storage unit 205 and a single change quantity ordisplacement quantity for the icon 30A may be calculated based upon theplurality of reach positions thus stored.

In addition, the image generation unit 201 does not need to adjust thedisplay mode for the midair image 30 if the distance between the reachposition 50 and the detection reference 40 is equal to or less than apredetermined value, i.e., if the reach position 50 is close to thedetection reference 40.

Furthermore, the image generation unit 201 does not need to adjust thedisplay mode for the icon 30A through each session of the calibrationprocessing. Instead, the control unit 20 may calculate the number oftimes an operation at the icon display position has failed based uponthe number of times that the reach position 50 has been determined andthe number of times the reach position 50 has been judged to actuallyreach the detection reference 40 through a plurality of sessions of thecalibration processing. The image generation unit 201 may adjust thedisplay mode for the icon 30A only if the number of times a failure hasoccurred is judged to be equal to or greater than a predetermined value.

While the calibration processing in this variation is adopted in thefirst calibration processing mode in the first embodiment in the exampledescribed above, the calibration processing may also be adopted in thesecond calibration processing mode and in any of variations 1 through 3.

In the calibration processing executed in variation 4 described above,the control unit 20 determines the reach position 50 by detecting anatural operating motion that the user would normally make whenperforming an operation at the display position of an icon 30A in amidair image 30. Namely, the control unit 20 determines the furthestreach position 50 by detecting a downward movement of the finger as ifto press down on the icon, which then shifts to an upward movement, ordetecting a movement of the finger as if to come into contact with theicon, and then hold down the icon briefly. Thus, the calibrationprocessing can be executed without the user being aware of the reachposition 50 being detected and/or determined through the calibrationprocessing, i.e., without the user being aware that calibrationprocessing is in progress.

(Variation 5 of the First Embodiment)

In the first embodiment, the operation detector 13 determines the reachposition 50 by detecting an operation the user performs with his fingerat the display position of the midair image 30 and the image generationunit 201 adjusts the display mode for the midair image 30 based upon thereach position 50. As an alternative, the user may be allowed to specifythe finger position at which he has experienced a perception ofperforming an operation at the display position of an icon in the midairimage, and in such a case, the detection reference control unit 204 mayrecognize the specified position and the image generation unit 201 mayadjust the display mode for the icon 30A based upon the specifiedposition. The following is a description of a variation in which theuser indicates the position at which he has had a perception ofperforming an operation at the display position of the midair image as aspecified position. It is to be noted that while an example in whichvariation 5 is adopted in the first calibration processing mode in thefirst embodiment is described below, it may also be adopted in thesecond calibration processing mode and in variations 1 through 4described earlier.

The following is a description of the display device in variation 5. Asthe display device 1 is started up and the user operates the calibrationprocessing mode selector operation button to select the firstcalibration processing mode, the calibration unit 203 in FIG. 2 startsthe first calibration processing mode. The image generation unit 201generates display image data, and the display unit 11 brings up adisplay image to be used in the calibration processing based upon thedisplay image data. FIG. 22 shows the display image generated for thecalibration processing brought up as a midair image 300. The midairimage 300 includes an icon 300B for calibration, and a message “Point atthis icon with finger and move the finger sideways for calibration” issuperimposed on the calibration icon 300B. In addition, the detectionreference control unit 204 sets the detection reference 40 to an initialposition near the midair image 300, as indicated in FIG. 23(a).

The user, following the instructions in the message superimposed on theicon 300B, moves his fingertip F down toward the icon 300B, as shown inFIG. 23(a), in order to perform an operation at the display position ofthe icon 300B. As the fingertip F reaches the electrostatic capacitancedetection range 13A of the operation detector 13 shown in FIG. 2, theoperation detector 13 detects the movement of the user's fingertip Ftoward the icon 300B, i.e., the downward movement, as a change in theelectrostatic capacitance.

The user moves his finger further downward and upon feeling that thefingertip F has reached the display position of the icon 300B in themidair image 300, he moves the finger F sideways, as indicated by thearrow in FIG. 23(b). The operation detector 13 detects the downwardmovement and the lateral movement of the finger F. The detectionreference control unit 204 designates the heightwise position of thefinger F at the time point at which it determines that the downwardmovement has switched to lateral movement as the operation detector 13,having detected a downward movement of the finger thus far, detects alateral movement of the finger F, as a specified position 50A. The imagegeneration unit 201 adjusts the display mode for the icon 300B, i.e.,calculates a change quantity with respect to the size of the icon 300Band a displacement quantity with respect to its display position, basedupon the specified position 50A. Data indicating the change quantity andthe displacement quantity for adjustment are stored into the storageunit 205. It is to be noted that while the specified position 50A islocated further upward relative to the midair image 300 in the examplepresented in FIG. 23(b), the specified position 50A, i.e., the positionat which the user experiences a perception as if his fingertip F hasreached the icon 300B in the midair image 300, may be in alignment withthe midair image 300 or may be further downward relative to the midairimage 300.

It is to be noted that the detection reference control unit 204designates the heightwise position taken by the finger when the downwardmovement of the finger F has shifted to the lateral movement as thespecified position 50A in the description provided above, the presentinvention is not limited to this example. The detection referencecontrol unit 204 may instead designate the height of the finger F at theend of the lateral movement following the downward movement as thespecified position 50A. As a further alternative, the detectionreference control unit 204 may designate the average or the median ofthe heights of the finger F assumed during the period of time elapsingbetween the start of the lateral movement of the finger F and the end ofthe lateral finger movement as the specified position 50A. As describedabove, the specified position 50A, at which the operation has beendetected, is detected by the detection reference control unit 204.

In reference to the flowchart presented in FIG. 24, the calibrationprocessing executed in variation 5 will be described. It is to be notedthat the flowchart in FIG. 24 only shows the processing executed in stepS121 through step S129 and does not show the processing executed insubsequent steps. The processing executed in step S129 and subsequentsteps is similar to the processing executed in step S109 and subsequentsteps in the flowchart presented in FIG. 21.

The processing executed in step S121 through step S124 is similar tothat executed in step S1 through step S4 in the flowchart presented inFIG. 10. In step S126, the operation detector 13 detects the lateralmovement of the user's finger. In step S127, the detection referencecontrol unit 204 decides, based upon the detection output from theoperation detector 13, that a shift has occurred in the movement of thefinger F from the downward movement to the lateral movement, designatesthe position taken by the finger F at the time of the shift as thespecified position 50A. The image generation unit 201 adjusts thedisplay mode for the icon 300B based upon the specified position 50A andstores data indicating the change quantity for the icon size and thedisplacement quantity for the icon display position into the storageunit 205 before the operation proceeds to step S128. In step S128, thefirst calibration processing mode ends and the operation proceeds tostep S129. In step S129, the midair image operation mode starts. In themidair image operation mode, the image generation unit 201 adjusts thedisplay mode for the icon 300B based upon the data indicating the changequantity and the displacement quantity read out from the storage unit205.

It is to be noted that while the processing executed in the firstcalibration processing mode is explained above, processing will beexecuted in the second calibration processing mode as shown in theflowchart presented in FIG. 12, in reference to which the firstembodiment has been described. However, if the detection referencecontrol unit 204 detects that the finger F, having been moving downward,has switched to a lateral movement through detection of an operationperformed by the user with respect to the icon 30A in step S24 in theflowchart presented in FIG. 12, the specified position 50A is determinedinstead of the reach position in step S27 and step S30. If the specifiedposition 50A is determined in step S30, the detection reference controlunit 204 makes a decision in step S30 as to whether or not the specifiedposition 50A is in alignment with the position of the detectionreference 40.

While the calibration processing in variation 5 is in progress, the userspecifies a position at which he experiences a perception of havingperformed an operation at the display position of the midair image 300with his finger F by switching the movement of his finger F from adownward movement to a lateral movement. In other words, the calibrationprocessing is executed by the display device 1 by allowing the user tospecify a position perceived as the display position operation positionwith respect to the icon 300B and thus, accurate calibration processingis enabled. In addition, indicating the specified position by switchingthe movement of the finger F from a downward movement to a lateralmovement assures good operability and the calibration processing can beexecuted promptly.

(Variation 6 of the First Embodiment)

The user operating the display device 1 in variation 5 indicates theposition at which he experiences a perception of performing an operationat the icon display position with his finger as a specified position byswitching his finger movement from a downward movement to a lateralmovement. The user of the display device 1 in variation 6 indicates theposition at which he experiences a perception of performing an operationat the icon display position with his finger by operating another icon.The calibration processing executed in this variation will be describednext. It is to be noted that while an example in which variation 6 isadopted in the first calibration processing mode in the first embodimentis described below, it may also be adopted in the second calibrationprocessing mode and in variations 1 through 5 described earlier.

As the display device 1 is started up and the user operates thecalibration processing mode selector operation button to select thefirst calibration processing mode, the calibration unit 203 in FIG. 2starts the first calibration processing mode. The image generation unit201 generates display image data, and the display unit 11 brings up adisplay image to be used in the calibration processing based upon thedisplay image data. A midair image 300 brought up in this situationincludes the icon 300B for calibration, shown in FIG. 22 in reference towhich variation 5 of the first embodiment has been described, andanother icon displayed near the icon 300B (e.g., to the left in thedrawing). A message “Touch the icon on the left side with a finger ofyour left hand while pointing to this icon with a finger of your righthand for calibration”, instead of the message shown FIG. 22, issuperimposed on the calibration icon 300B.

The user, following the instructions in the message superimposed at theicon 300B, moves a fingertip F of his right hand down toward the icon300B in order to perform an operation at the display position of theicon 300B. As the fingertip reaches the electrostatic capacitancedetection range 13A of the operation detector 13, the operation detector13 detects the movement of the user's finger moving closer to thedisplay position of the icon 300B, i.e., the downward movement of thefinger, as a change in the electrostatic capacitance. The user moves hisfinger further downward and as soon as he experiences a perception ofthe fingertip F performing an operation at the display position of theicon 300B in the midair image 300, he moves a fingertip of his left handtoward the other icon displayed near the icon 300B in order to performan operation at the display position of the other icon with thefingertip F of his left hand as directed in the message. The operationdetector 13 detects the movement of the fingertip F toward the othericon. The detection reference control unit 204 designates the positiontaken by the fingertip in the user's right hand at the time point atwhich the operation detector 13 detects that the user's finger ispositioned on the other icon as a specified position 50A. The imagegeneration unit 201 adjusts the display mode for the icon 300B bycalculating, based upon the specified position 50A, a change quantity bywhich the size of the icon 300B is to be altered and a displacementquantity by which the display position is to move as it does in thefirst embodiment. The image generation unit 201 then stores dataindicating the change quantity for the icon size and the displacementquantity for the icon display position having been calculated into thestorage unit 205.

It is to be noted that since the position taken by the right hand fingerwhen the user experiences a perception of performing an operation at thedisplay position of the right hand-side icon 300B is designated as thespecified position, the right hand finger needs to move down toward themidair image 300. However, the left hand finger only needs to bepositioned above or below the other icon to perform an operation at thedisplay position of the other icon on the left hand-side and thus, it isnot strictly necessary to move the left hand finger downward. Namely,the left hand finger may move along, for instance, a direction parallelto the plane of the midair image 300, i.e., along a lateral direction,until it reaches a point above or below the other icon.

Furthermore, it is not essential that a left hand finger and a righthand finger be used, as long as the operations described above can bedetected both on the icon 300B and on the other icon in the calibrationmidair image 300. For instance, these operations may be performed byusing two fingers of one hand on either side. In addition, instead ofperforming an operation at the display position of the other icon, theuser may press an OK button (not shown) at the display device 1 invariation 6.

Moreover, instead of designating the position taken by the right handfingertip when the user performs an operation at the display position ofthe other icon or when the user presses the OK button (not shown) as thespecified position, the position of the right hand fingertip assumedwhen the user makes a predetermined gesture with his left hand may bedesignated as the specified position. In such a case, the display device1 will include an image-capturing device 18 in variation 9 to bedescribed later (see FIGS. 26 and 27) so that a user gesture (e.g., thehand switching from the sign “stone” to the sign “paper”) is detected byusing images obtained via the image-capturing device 18.

In reference to the flowchart presented in FIG. 24 pertaining tovariation 5 of the first embodiment, the calibration processing executedin the variation will be described. The following explanation will focuson the primary differences from the processing having been described andan explanation of steps in which similar processing is executed is notprovided. In step S123 in FIG. 24, the icon 300B and the other icon arebrought up on display, and in step S124, the operation detector 13starts detection of a downward movement of the fingertip in the user'sright hand. The user moves his finger further downward and then performsan operation at the display position of the other icon with his lefthand fingertip at the time point at which he experiences a perception ofthe right hand fingertip performing an operation at the display positionof the icon 300B in the midair image 300. In step S127, the positiontaken by the right hand fingertip at the time point at which the userhas performed an operation at the display position of the other iconwith his left hand is designated as the specified position 50A and theimage generation unit 201 adjusts the display mode for the icon 300Bbased upon the specified position 50A. At this time, too, dataindicating the size change quantity and the display positiondisplacement quantity for adjustment are stored into the storage unit205.

It is to be noted that while the processing executed in the firstcalibration processing mode is explained above, processing will beexecuted in the second calibration processing mode as shown in theflowchart presented in FIG. 12, in reference to which the firstembodiment has been described. However, in step S24 in the flowchartpresented in FIG. 12, the detection reference control unit 204 detectsan operation performed by the user with respect to the icon 30A by usinghis right hand. While the user is performing an operation for the icon30A with his right hand, the detection reference control unit 204 maydetect an operation performed by the user using his left hand at thedisplay position of the other icon. In such a case, the detectionreference control unit 204 determines that the position of the fingertipof the user's right hand is the specified position 50A, instead ofdetermining the reach position, in step S27 and step S29. In step S30, adecision is made as to whether or not the specified position 50Adetermined in step S29 is in alignment with the detection reference 40.

In variation 6, the user indicates a specified position at which thefinger operates the icon during the calibration processing by operatinganother icon or by operating the OK button at the display device 1. Thecalibration processing allowing the user to specify the position atwhich he perceives the icon 300B can be executed with high accuracy inthe display device 1. In addition, by allowing the user to indicate thespecified position through an operation at another icon or at a buttonat the display device, the calibration processing can be executedpromptly.

(Variation 7 of the First Embodiment)

The user of the display device in variation 7, perceiving that he hasperformed an operation at the display position of an icon with hisfingertip, indicates a specified position by holding the finger stillfor a predetermined length of time. It is to be noted that while anexample in which variation 7 is adopted in the first calibrationprocessing mode in the first embodiment is described below, it may alsobe adopted in the second calibration processing mode and in variations 1through 6 described earlier.

In this variation, a message “Point at this icon and hold the fingerstill for a moment for calibration” is brought up in a superimposeddisplay in an icon included in the calibration midair image. The user,perceiving that he has performed an operation at the icon displayposition, briefly holds the finger still and, in response, the operationdetector 13 detects cessation of the downward movement of the fingerover a predetermined length of time. The detection reference controlunit 204 designates the position at which the finger is held still asthe specified position based upon the detection output provided by theoperation detector 13 at this time.

The specified position is determined as described below. Namely, it isdecided that an operation has been performed at the display position ofan icon 300A when the fingertip F, having been moving downward, comes toa stop and is held still within a relatively small predetermined holdingrange taken along the up/down direction over a length of time equal toor greater than a predetermined time length. It is decided that anoperation has been performed at the display position of the icon 300Awith the fingertip F when the fingertip F stays within the specificholding range over the predetermined time length or longer, as describedabove, based upon the following rationale. Namely, the user operation atthe display position of the icon 300A in the midair image 300 isdifferent from an operation performed at a touch panel and the fingertipF may not become completely still at the display position of the icon300A. It is to be noted that the predetermined holding range inreference to which the specified position is determined will be set to avalue sufficiently small relative to the electrostatic capacitancedetection range 13A of the operation detector 13, e.g., 5 mm, and thepredetermined time length will be set to, for instance, 2 sec.

In variation 7, the user specifies a position at which he performs anicon operation with his finger by holding the fingertip F still duringthe calibration processing. Since the user is able to specify theposition at which he perceives the icon 300A to be located, the displaydevice 1 is able to execute accurate calibration processing.

(Variation 8 of the First Embodiment)

The user of the display device in variation 8 indicates with his voicethe specified position at which he experiences a perception ofperforming an operation with his fingertip at an icon display position.It is to be noted that while an example in which variation 8 is adoptedin the first calibration processing mode in the first embodiment isdescribed below, it may also be adopted in the second calibrationprocessing mode and in variations 1 through 7 described earlier.

FIG. 25 is a block diagram showing the control unit 20, and the displayunit 11 and an operation detector 13 controlled by the control unit 20,among the structural components in the display device 1 in thisvariation. The display device 1 includes a sound collector 14, with asound detection unit 208 installed in the control unit 20. The soundcollector 14 collects sound around the display device 1 and outputs thecollected sound as audio data to the sound detection unit 208. The soundcollector 14 may be a commonly available microphone. The sound detectionunit 208 designates the audio data provided from the sound collector 14and makes a decision as to whether or not the audio data express theword “yes”.

After the calibration unit 203 in FIG. 25 starts up the firstcalibration processing mode, the image generation unit 201 generatesdisplay image data, and the display unit 11 brings up a display image tobe used in the calibration processing based upon the display image data.A midair image 300 brought up on display in this situation includes thecalibration icon 300B shown in FIG. 22 in reference to which variation 5of the first embodiment has been described, and a message “Touch thisicon and say yes for calibration”, instead of the message in FIG. 22, issuperimposed on the calibration icon 300B.

The user, following the instructions in the message brought up in thesuperimposed display at the icon 300B, movies his fingertip down towardthe icon 300B in order to perform an operation at the display positionof the icon 300B. The user says “yes” as directed in the message when heperceives that his fingertip has touched the icon 300B. The operationdetector 13 detects the downward movement of the fingertip, and thesound collector 14 picks up the user's voice and outputs it as audiodata to the sound detection unit 208. As the sound detection unit 208decides that the audio data correspond to “yes”, the detection referencecontrol unit 204 designates the position taken by the fingertip,detected by the operation detector 13 at the exact time point, as aspecified position 50A. The image generation unit 201 adjusts thedisplay mode for the icon 300B by calculating a change quantity by whichthe size of the icon 300B is to be altered and a displacement quantityby which the display position is to move as it does in the firstembodiment. The image generation unit 201 then stores data indicatingthe change quantity and the displacement quantity having been calculatedinto the storage unit 205.

The calibration processing described above will be explained inreference to the flowchart presented in FIG. 24 pertaining to variation5 of the first embodiment. Since only the calibration processingexecuted in step S126 in FIG. 24 is distinguishable from that havingbeen explained earlier, the explanation will focus on step S126 in FIG.24 and a repeated execution of the processing executed in other stepswill not be provided. In step S126 in FIG. 24, the sound detection unit208 makes a decision as to whether or not the user has said “yes” basedupon the output from the sound collector 14. If an affirmative decisionis made in step S126, i.e., if it is decided that the user, perceivingthat he has touched the icon 300B, has said “yes”, the detectionreference control unit 204 designates the position of the fingertipassumed at the time point at which the sound detection unit 208recognizes the word “yes” as the specified position 50A, i.e.,determines it to be the specified position 50A.

It is to be noted that while the processing executed in the firstcalibration processing mode is explained above, processing will beexecuted in the second calibration processing mode as shown in theflowchart presented in FIG. 12, in reference to which the firstembodiment has been described. However, if a user operation performedwith respect to the icon 30A is detected in step S24 in the flowchartpresented in FIG. 12 and the sound detection unit 208 recognizes theword “yes”, the detection reference control unit 204 designates thespecified position 50A instead of determining the reach position in stepS27 and step S29. In step S30, a decision is made as to whether or notthe specified position 50A designated in step S29 is in alignment withthe detection reference 40.

In variation 8, the user vocally indicates the specified position takenby his finger when he perceives that an operation has been performed atthe display position of the icon 300B. By allowing the user to indicatethe reach position with his voice, the display device 1 is able toexecute the calibration processing promptly.

It is to be noted that the display device 1 does not need to include thesound collector 14 and the sound detection unit 208 in such aconfiguration may execute sound detection by using audio data obtainedat an external sound collecting device and input from the external soundcollecting device via either a wireless means or a wired means.

(Variation 9 of the First Embodiment)

While the downward movement of the user's fingertip is detected by theoperation detector 13 configured with a capacitive panel in thedescription provided above, the position of the user's fingertip may bedetected by an image-capturing device, instead. The display device 1 invariation 9 includes an image-capturing device (e.g., a digital camera)18 to function as an operation detector, disposed at the upper surfaceof the display device 1, as shown in FIG. 26(a). A block diagrampertaining to such a display device 1 is provided in FIG. 26(b).

The control unit 20 of the display device 1 in the block diagrampresented in FIG. 26(b) includes an image analysis unit 210. Theimage-capturing device 18 captures an image of an object located abovethe display unit 11, i.e., the user's finger, and the captured image isinput to the image analysis unit 210. The image analysis unit 210determines the position of the user's fingertip by analyzing thecaptured image input from the image-capturing device 18. Namely, theimage analysis unit 210 makes a decision based upon the position of theimage of the finger within the captured image, with respect to aspecific icon, among the plurality of icons, being operated with theuser's fingertip. In addition, the image analysis unit 210 compares thesize of the finger image within the captured image with a standardfinger size, and more specifically, with the size of a finger at apredetermined heightwise position, an image of which has been capturedin advance, so as to determine the heightwise position of the finger,i.e., the position taken by the descending finger. Through this process,the position of the user's fingertip within the three-dimensional spacecan be determined. The display device 1 in variation 9 structured asdescribed above is capable of obtaining, through analysis of thecaptured image provided via the image-capturing device 18, informationsimilar to the information pertaining to the fingertip position obtainedvia the operation detector 13 configured with a capacitive panel. Thus,the display device in variation 9 is able to execute processing similarto that executed in the embodiment and the variations thereof describedearlier, by using the image-capturing device 18 instead of thecapacitive panel having been described in reference to the embodimentand the variations 1 through 8 thereof.

While the image analysis unit 210 in the display device 1 in variation 9calculates the heightwise position of the finger based upon the size ofthe finger in the captured image, the image-capturing device 18 mayinstead detect the heightwise position of the finger via a phase focusdetection device and an image recognition device mounted in the digitalcamera. In more specific terms, the image recognition device mayrecognize a finger, the phase focus detection device may detect adefocus quantity with respect to the finger recognized by the imagerecognition device and the heightwise position of the finger may becalculated based upon the defocus quantity. Furthermore, the heightwiseposition of the finger may be likewise detected via a contrast focusdetection device that may be mounted in the digital camera instead ofthe phase focus detection device.

It may be ideal to configure the image-capturing device 18 with a camerahaving installed therein a TOF (time of flight) device instead of aphase focus detection device or a contrast focus detection device. A TOFcamera emits infrared radiation from the camera body, receives infraredlight that is reflected off a target object and then enters the TOFcamera, and calculates the distance from the TOF camera to the targetobject based upon a phase change having occurred in the received lightrelative to the emitted light. Accordingly, by designating the user'sfingertip as the measurement target object, emitting infrared light fromthe TOF camera toward the user's fingertip and receiving light reflectedfrom the fingertip, the distance from the TOF camera to the user'sfingertip can be determined. It is desirable that the image-capturingdevice 18 include an image-capturing lens constituted with a wide-anglelens so as to cover the entire midair image 30 and such animage-capturing lens may be a fisheye lens. In addition, the displaydevice may include a plurality of image-capturing devices (e.g., twoimage-capturing devices) and the position of the user's fingertip may bedetected based upon captured images provided from the plurality ofimage-capturing devices.

FIG. 27 presents an example of a display device 1 equipped with a TOFcamera. FIG. 27 simply shows the internal structure of the displaydevice 1 and does not provide an illustration of the display devicebody. As FIG. 27 shows, an installation space for a TOF camera 118′ isformed so as to take up a position corresponding to the centers of adisplay unit 11 and an image-forming optical element 12, and the TOFcamera 118′ is disposed in this installation space. The TOF camera 118′radiates infrared light onto the user's fingertip by scanning infraredlight over a predetermined range and measures the distance from the TOFcamera 118′ to the user's fingertip based upon a change in the phase ofreflected light. Based upon the distance and the infrared emissiondirection, the position of the user's fingertip in the three-dimensionalspace relative to the TOF camera 118′ can be determined. In other words,the specific position within the midair image plane corresponding to thefingertip position and the distance that sets the fingertip positionapart from the surface of the display device 1 can be determined.Information similar to detection information indicating the fingertipposition obtained in conjunction with a capacitive panel can thus beobtained based upon the range-finding results provided by the TOF camera118′. It is to be noted that while an installation space for the TOFcamera 118′ is formed so as to take up an area corresponding to thecenters of the display unit 11 and the image-forming optical system 12and the TOF camera 118′ is disposed in this space in the descriptionprovided above, the present invention is not limited to this example andit may be adopted in a configuration that includes a TOF camera 118′installed outside the display unit 11.

At the display device 1 in variation 9, too, a midair image 30 is formedat a position above the image-forming optical system 12 of the displaydevice 1, set apart from the image-forming optical system 12 by adistance H1, and the detection reference 40 is set at a position abovethe image-forming optical system 12, set apart from the image-formingoptical system 12 by a distance H2 (H1<H2), as shown in FIG. 27. The TOFcamera 118′ assumes a detection range 13A for detection of the user'sfingertip position, set further upward relative to the surface of theimage-forming optical system 12. In FIG. 27, the limit to the range overwhich images can be captured is indicated by a dotted line 13 a abovethe TOF camera 118′ and the detection range 13A is defined by thedetection limit 13 a and the surface of the image-capturing opticalsystem 12. In variations 9, too, the midair image 30 and the detectionreference 40 are set inside the detection range 13A, as in the firstembodiment and variations 1 through 8 described earlier. It is to benoted that while the detection reference 40 in FIG. 27 is set furtherupward relative to the midair image 30, it may instead be set furtherdownward relative to the midair image 30 or in alignment with theposition of the midair image 30, as long as it is set within thedetection range 13A. In addition, a range other than the zone set as thedetection reference 40 within the detection range 13A is referred to asa detection reference outside range 41 in the description of variation9. It is to be noted that instead of setting the detection range 13A bydefining the limit to the range over which an image can be captured withthe TOF camera 118′, the detection range 13A may be set as a range madeup with part of the range over which an image can be captured by takingoff part of the image-capturing enabled range (e.g., predeterminedranges at the left and the right ends in FIG. 27).

The display device 1 in variation 9 described above includes animage-capturing device 18 instead of an operation detector 13 configuredwith a capacitive panel. However, the display device 1 may include bothan operation detector 13 and an image-capturing device 18. In such acase, the detection range 13A of the operation detector 13 shown in FIG.3 may be divided into, for instance, two parts, i.e., a top part and abottom part, so as to form a lower detection range (a detection rangecloser to the display unit 11) and an upper detection range (a detectionrange further away from the display unit 11), and the lower detectionrange and the upper detection range may be respectively designated asthe detection range for the capacitive panel and as the detection rangefor the image-capturing device 18. In this configuration, as the usermoves his finger downward in order to perform an operation at thedisplay position, the image-capturing device 18 detects the first halfof the descending movement of the finger and the capacitive paneldetects the second half of the descending movement of the finger.Generally speaking, highly accurate detection is enabled via thecapacitive panel over a range set above and in close proximity to thedisplay unit 11, whereas it may not always be possible to capture animage with the image-capturing device 18 over a range set above and invery close proximity to the display unit 11. For this reason, it isdesirable to assign different detection ranges to the capacitive paneland the image-capturing device 18 as described above. It is to be notedthat the detection range 13A does not need to be divided into two equalparts along the up/down direction and instead, it may be divided intoparts that are not equal. In addition, an operation detector 13configured with another device, such as a proximity sensor, instead ofthe capacitive panel or the image-capturing device 18, may be used. Thismeans that detection ranges formed by dividing the detection range 13Amay be assigned to various operation detectors 13.

The velocity·acceleration detection unit 206 shown in FIG. 19 is alsocapable of calculating the velocity and the acceleration with which thefinger moves based upon a captured image provided by the TOF camera 118′in FIG. 27. Accordingly, in correspondence to each of the upper andlower detection ranges formed by dividing the detection range 13A, thefinger movement velocity or the finger movement acceleration may becalculated so as to enable the reach position predicting unit 207 topredict the reach position of the finger.

In addition, instead of the image-forming optical system 12 or 12A,having been explained in reference to the first embodiment andvariations 1 through 9 thereof, an image-forming optical systemconfigured with a half mirror and a retro-reflective member may be used.A retro-reflective member may be constituted with a reflecting memberthat includes, for instance, a plurality of three-dimensional rightangle triangular pyramid prisms and reflects light having enteredtherein back along the same optical path. In a structure that includessuch a component, light having departed the display unit 11 is reflectedat the half mirror (or transmitted through the half mirror), then entersthe retro-reflective member and is then reflected along the same opticalpath. The light reflected at the retro reflective member forms an imageas it advances through the same optical path. The light having departedthe retro-reflective member re-enters the half mirror, is transmittedthrough the half mirror (or is reflected at the half mirror) and forms amidair image by forming an image at a position conjugate with thedisplay unit 11.

It is to be noted that while the display device 1 in the firstembodiment and its variations 1 through 9 described above includes atleast the control unit 20, the display unit 11 and the operationdetector 13, the present invention may instead be adopted in a controldevice configured with the control unit 20 alone or a control deviceconfigured with the control unit 20 and the operation detector 13. Inaddition, the control unit 20 only needs to include, at least, thecalibration unit 203 and the image generation unit 201. A structuralelement among the structural elements described above may be added asneeded in order to realize the various advantages described in referenceto the first embodiment or any of variations 1 through 9. In addition,the control device described above may be built into any of varioustypes of electronic devices adopting the first embodiment and thevariations thereof.

Furthermore, the present invention may be adopted in a detection deviceconfigured with the control unit 20 alone or a detection deviceconfigured with the control unit 20 and the operation detector 13.Moreover, the control unit 20 only needs to include at least thecalibration unit 203 and the image generation unit 201. In order toenable such a detection device to achieve the various advantagesdescribed in reference to the first embodiment or variations 1 through9, a structural element among the structural elements described earliermay be added into the detection device as deemed necessary.

Second Embodiment

In reference to drawings, a display device 1 in the second embodimentwill be described. The second embodiment will be described in referenceto an example in which the display device 1 in the embodiment is mountedin an operation panel. It is to be noted that the display device 1 inthe embodiment does not need to be mounted in an operation panel and maybe mounted in any type of electronic apparatus as has been explained inreference to the first embodiment and variations thereof.

The display device 1 in the second embodiment may adopt a structuresimilar to that of the display device 1 in the first embodiment shown inFIG. 1 and FIG. 2. It is to be noted that it may include an imagecapturing device 18 as does the display device 1 in variation 9 of thefirst embodiment illustrated in FIG. 26. In addition, any of the variousmethods described in reference to the first embodiment and thevariations of the first embodiment may be applied to the detection ofthe reach position or the specified position and a display mode for amidair image 30 in the display device 1 in the second embodiment.

Even after the display device 1 adjusts the display mode for an icon300A, as has been described in reference to the first embodiment and thevariations thereof, the operation detector 13 still may not be able todetect a user operation at the detection reference 40, since the icon300A is displayed as part of a midair image 300 and even after thedisplay mode is adjusted, a given user may perceive the display positionof the icon 300A differently from another user. For instance, after thedisplay mode is adjusted so as to create a user perception as if theicon 300A has moved further away from the user, a user may perceive thatthe icon 300A has moved by an extent equal to or greater than the extentof displacement intended to be perceived through the display modeadjustment. Under such circumstances, the user may perform an operationwith respect to the icon 300A by an extent greater than the extent ofchange in the user operation expected to occur as a result of thedisplay mode adjustment and the fingertip F of the user may thus bepositioned further downward (toward the Z direction − side) than thedetection reference 40. In another example, even after the display modefor the icon 300A is adjusted, the user may not perceive that the icon300A has moved at all. Under such circumstances, the user may perform anoperation for the icon 300A in the same way as he has done before thedisplay mode adjustment for the icon 300A and thus the fingertip F ofthe user may not reach the detection reference 40 or may repeatedly passthrough the detection reference 40.

The display device 1 in the second embodiment further adjusts thedisplay mode for the icon 300A in such a situation so as to enabledetection of a user operation at the detection reference 40. Theprocessing executed when the user operation changes to an excessiveextent after the display mode for the icon 300A is adjusted and theprocessing executed when no change occurs in the user operation evenafter the display mode for the icon 300A is adjusted will beindividually explained below.

The following is an explanation of the calibration processing executedin the embodiment in the first calibration processing mode.

<When the User Operation Changes to an Excessive Extent after theDisplay Mode for an Icon 30A is Adjusted>

FIG. 28(a) is a schematic illustration presenting an example of a midairimage 300 that may be displayed by the display device 1 in thisembodiment. The midair image 300 in FIG. 28(a) is brought up in aninitial display, as is the midair image 300 shown in FIG. 4(a)pertaining to the first embodiment. Even after the user performs anoperation for an icon 300A (300A0) in the midair image 300 brought up insuch an initial display, the reach position 50 may still be locatedabove the detection reference 40, as shown in FIG. 7(b).

In the calibration processing executed under such circumstances, theimage generation unit 201 adjusts the display mode for the icon 300A soas to create a user perception as if the icon 300A has moved furtheraway from the user (toward the Z direction − side). In this example, theimage generation unit 201 alters the display position of the icon 300Aas a way of adjusting the display mode for the icon 300A. The imagegeneration unit 201 moves the display position of the icon 300A closerto the vanishing point, as illustrated in FIG. 28(b). It is to be notedthat the displacement quantity representing the extent to which thedisplay position of the icon 300A is to move, is determined based uponthe distance between the detection reference 40 and the reach position50, as in the first embodiment. The processing executed as describedabove will be referred to hereafter as first processing, and theadjusted display mode for the icon 300A1 resulting from the firstprocessing will be referred to as a first display mode.

The user performs a press-down operation with respect to the icon 300A1having been switched to the first display mode, so that his fingertip Fis positioned further toward the Z direction − side in comparison to itsposition in the first processing. At this time, if the user perceives asif the icon 300A on display in the first display mode has moved alongthe depthwise direction (toward the Z direction − side) by an excessivedegree, the fingertip F of the user, i.e., the reach position 50, willbe set further downward relative to the detection reference 40, as shownin FIG. 7(d).

In this situation, the image generation unit 201 adjusts the displaymode for the icon 300A1 so as to create a user perception as if the icon300A has moved closer to the user (toward the Z direction+ side). Theimage generation unit 201 moves the display position of the icon 300Afurther away from the vanishing point, as illustrated in FIG. 28(c). Itis desirable that the image generation unit 201 set the displacementquantity for the icon 300A to a value smaller than, e.g., approximately75%, of the value representing the displacement quantity for the icon300A1 determined through the first processing. The rationale for this isthat if the icon 300A is displayed in the state prior to the firstprocessing, i.e., in the state shown in FIG. 28(a), the user operation,too, may resume the state shown in FIG. 7(b). The processing executed asdescribed above will be referred to hereafter as second processing andthe adjusted display mode for the icon 300A2 resulting from the secondprocessing will be referred to as a second display mode.

It is to be noted that while the image generation unit 201 sets thedisplacement quantity for the icon 300A2 to a value smaller than thatrepresenting the displacement quantity for the icon 300A1 determinedthrough the first processing, it is not strictly necessary to set asmaller value than that corresponding to the first processing. Forinstance, the image generation unit 201 may select a displacementquantity matching the displacement quantity determined in the firstprocessing and thus display the icon 300A2 at a position matching thedisplay position shown in FIG. 28(a). In addition, the image generationunit 201 may select a greater displacement quantity for the icon 300A2relative to the displacement quantity determined in the first processingso as to display the icon 300A2 at a position further toward the Ydirection+ side relative to the display position shown in FIG. 28(a).The rationale for the image generation unit 201 to execute processing asdescribed above is that in this embodiment, a user perception is createdas if the position of the icon 300A, taken along the Z direction, haschanged by adjusting the display mode. This means that with respect tothe position of the icon 300A along the Z direction, the depthwiseposition perceived when the display position of the icon 300A has beenmoved further away and the depthwise position perceived when the displayposition of the icon 300A has been moved closer may be different fromeach other (or they may be perceived as different positions dependingupon the user). For this reason, there may be circumstances under whichit is desirable to set the displacement quantity for the secondprocessing equal to or greater than the displacement quantity for thefirst processing.

As the display mode for the icon 300A2 is adjusted to the second displaymode through the second processing, a user perception as if the icon300A has moved closer to the user is created. As a result, the user,realizing at the second processing that he has performed an excessivepress-down operation with respect to the icon 300A, is expected torefrain from subsequently performing a second excessive press-downoperation toward the Z direction − side.

In the explanation provided above, the user performs a press-downoperation for the icon 300A in FIG. 28(a) above the detection reference40. If the user has performed a press-down operation under the detectionreference 40, the image generation unit 201 needs to execute processingthe reverse of that described above. Namely, during the firstprocessing, the image generation unit 201 adjusts the display mode so asto create a user perception as if the icon 300A1 has moved closer to theuser (the Z direction+ side). The image generation unit 201 switches tothe first display mode through the first processing by moving thedisplay position of the icon 300A1 further away from the vanishingpoint.

The user performs a press-down operation with respect to the icon 300A1having been switched to the first display mode, so that his fingertip Fis positioned further toward the Z direction+ side in comparison to itsposition during the first processing. At this time, if the userperceives as if the icon 300A on display in the first display mode hasbeen moved closer to the user (toward the Z direction − side) by anexcessive degree, the fingertip F of the user, i.e., the reach position50, will be set above the detection reference 40, as shown in FIG. 7(a).In this situation, the image generation unit 201 adjusts the displaymode for the icon 30A1 so as to create a user perception as if the icon300A1 has moved further away from the user (toward the Z direction −side). The image generation unit 201 switches to the second display modein the second processing by moving the display position of the icon 300Atoward the vanishing point. As the display mode for the icon 300A2 isadjusted to the second display mode through the second processing, auser perception as if the icon 300A2 has moved further away from theuser is created. As a result, the user, recognizing that the press-downoperation having been executed in the first processing has not reachedthe icon 300A (detection reference 40), is expected to adjust the extentof a subsequent press-down operation performed toward the Z direction −side.

While an explanation has been given above on an example in which thedisplay mode for the icon 300A is adjusted by altering the displayposition, the present invention is not limited to this example. Thedisplay mode for the icon 300A may instead be adjusted by altering itssize, by altering any of the other parameters listed in reference tovariation 1 of the first embodiment or by altering any of these incombination. For instance, in order to create a user perception as ifthe display position of the icon 300A1 has moved further away from theuser through the first processing by altering the size of the icon 300A,the image generation unit 201 reduces the size of the icon 300A1 toswitch to the first display mode. In the second processing, the imagegeneration unit 201 switches to the second display mode by increasingthe size of the icon 300A2. In addition, in order to create a userperception as if the display position of the icon 300A1 has moved closerto the user through the first processing the image generation unit 201increases the size of the icon 300A1 to switch to the first displaymode. In the second processing, it switches to the second display modeby decreasing the size of the icon 300A2.

It is to be noted that the image generation unit 201 engaged in thesecond processing may first adjust the display mode so as to switch theicon 300A1 displayed in the first display mode back to the initialdisplay and may then adjust the display mode for the icon 300A to thesecond display mode. Namely, following the first processing, the imagegeneration unit 201 may switch the midair image 300 displayed as shownin FIG. 28(b) to the midair image 300 shown in FIG. 28(a). The imagegeneration unit 201 may then execute the second processing by switchingthe display to the midair image 300 shown in FIG. 28(c).

In addition, the display mode does not need to be switched back to theinitial display in the processing described above. For instance, theimage generation unit 201 may first adjust the display mode for the icon300A by selecting a display position corresponding to a displacementquantity amounting to, for instance, 75% of the displacement quantityrepresenting the full distance between the icon 300A1 and the icon300A0, and may then switch to the second display mode. As a furtheralternative, the image generation unit 201 may first adjust the displaymode for the icon 300A by selecting a display position corresponding toa displacement quantity amounting to, for instance, 125% of thedisplacement quantity representing the full distance between the icon300A1 and the icon 300A0, and then may switch to the second displaymode.

Through these measures, the user is made aware that he has performed anoperation to an excessive degree following the first processing and thusis expected to perform an operation at the position at which thedetection reference 40 is located in the midair image operation modeduring or after the second processing.

Furthermore, the image generation unit 201 may bring up on display anicon 300A2 in the second display mode while the icon 300A1 is still ondisplay in the first display mode. For instance, the image generationunit 201 may provide a display in the first display mode in a cold colorsuch as blue and may adjust the display mode by raising the luminancefor the second display mode. Under such circumstances, the imagegeneration unit 201 displays the icon 300A2 assuming a blue color andhigh luminance as a result of the switchover to the second display modeby raising the luminance of the blue icon 300A1. In this case, too, theuser is made aware that he has performed an operation to an excessivedegree following the first processing and thus is expected to perform anoperation at the position at which the detection reference 40 is locatedin the midair image operation mode during or after the secondprocessing.

<When the User Operation Remains Unchanged Even after the Display Modefor the Icon 300A is Adjusted>

An explanation is given below in reference to an example in which a useroperation is performed for an icon 300A (300A0) in a midair image 300brought up in an initial display, as shown in FIG. 29(a). It is to benoted that the midair image 300 shown in FIG. 29(a) is similar to themidair image 300 in FIG. 28(a). The reach position 50 may be locatedabove the detection reference 40, as shown in FIG. 7(b), with respect tothis midair image 300.

Under these circumstances, the image generation unit 201 executes thefirst processing in the same way as it executes processing when the useroperation changes to an excessive degree as described above. The imagegeneration unit 201 adjusts the display mode for the icon 300A so as tocreate a user perception as if the icon 300A has moved further away fromthe user (toward the Z direction − side). Namely, the image generationunit 201 switches to the first display mode by moving the displayposition of the icon 300A1 closer to the vanishing point, as illustratedin FIG. 29(b). It is to be noted that the displacement quantityrepresenting the extent to which the display position of the icon 300A1is to move is determined based upon the distance between the detectionreference 40 and the reach position 50, as in the first embodiment. Inaddition, the midair image 30 shown in FIG. 29(b) is similar to themidair image 300 in FIG. 28(b).

If the user does not perceive as if the icon 300A1 displayed in thefirst display mode has moved along the depthwise direction (toward the Zdirection − side), the user is likely to perform a press-down operationfor the icon 300A1 displayed in the first display mode resulting fromdisplay mode adjustment just as he has performed the press-downoperation during the first processing. Namely, the fingertip F of theuser is likely to be positioned above the detection reference 40, asillustrated in FIG. 7(b).

In this situation, in order to lead the user to perform a press-downoperation at a position closer to the detection reference 40 compared tothe current position, i.e., in order to set the reach position 50 at thedetection reference 40, the image generation unit 201 adjusts thedisplay mode for the icon 300A1 to the second display mode in the secondprocessing. Even after the display position of the icon 300A has beenaltered, the user has not perceived as if the icon 300A has moved andaccordingly, the image generation unit 201 switches to the seconddisplay mode by altering the size of the icon 300A2 in the secondprocessing. At this time, the image generation unit 201 reduces the sizeof the icon 300A2 as shown in FIG. 29(c) in the second processing, so asto create a user perception as if the display position of the icon 300A2has moved further away from the user. Namely, the image generation unit201 switches the display mode for the icon 300A to the second displaymode (related to the icon size) different from the first display mode(related to the icon display position) which has not elicited therequired action of the user. As a result, the user is made aware at thesecond processing that the press-down operation he has performed has notreached the icon 300A (detection reference 40) and is thus expected tosubsequently adjust the extent to which he performs the press-downoperation toward the Z direction − side.

In the explanation provided above, the user performs a press-downoperation for the icon 300A in FIG. 29(a) above the detection reference40. If the user has performed a press-down operation under the detectionreference 40, the image generation unit 201 needs to execute processingthat is the reverse of that described above. Namely, during the firstprocessing, the image generation unit 201 adjusts the display mode so asto create a user perception as if the icon 300A1 has moved closer to theuser (the Z direction+ side).

The image generation unit 201 switches to the first display mode throughthe first processing by moving the display position of the icon 300A1further away from the vanishing point. If the fingertip F of the user isstill positioned further downward relative to the detection reference 40following the first processing, the image generation unit 201 switchesto the second display mode by increasing the size of the icon 300A2through the second processing.

It is to be noted that while the image generation unit 201 alters thedisplay position of the icon 300A1 while the image generation unit 201switches to the first display mode by altering the display position ofthe icon 300A1 and switches to the second display mode by altering thesize of the icon 300A2 in the example described above, the presentinvention is not limited to this example. The image generation unit 201may instead switch to the first display mode by altering the size of theicon 300A1 and switch to the second display mode by altering the displayposition of the icon 300A2. Moreover, the image generation unit 201 mayswitch to the first display mode by altering one of the variousparameters listed in reference to variation 1 of the first embodimentand switch to the second display mode by altering another parameteramong the various parameters.

It is to be noted that in this case, too, the image generation unit 201may bring up on display the icon 300A2 in the second display mode whilethe icon 300A1 is still on display in the first display mode. Forinstance, the image generation unit 201 may provide a display in thefirst display mode in a cold color such as blue and may adjust thedisplay mode by lowering the luminance for the second display mode.Under such circumstances, the image generation unit 201 displays theicon 300A2 assuming a blue color and low luminance as a result of theswitch to the second display mode by lowering the luminance of the blueicon 300A1. As a result, the user is made aware that the operationperformed after the first processing has not been much different fromthe operation performed during the first processing and accordingly isled to perform an operation at the position at which the detectionreference 40 is located in the midair image operation mode during orafter the second processing.

In addition, the image generation unit 201 engaged in the secondprocessing may first adjust the display mode so as to switch the icon300A1 displayed in the first display mode back to the initial displayand then adjust the display mode for the icon 300A1 to the seconddisplay mode. Namely, following the first processing, the imagegeneration unit 201 may switch the midair image 300 displayed as shownin FIG. 29(b) to the midair image 300 shown in FIG. 29(a). The imagegeneration unit 201 may then execute the second processing by switchingto the midair image 300 shown in FIG. 29(c). Under these circumstances,the image generation unit 201 does not need to adjust the display modeso that the icon 300A1 displayed in the first display mode temporarilyresumes the initial display mode. For instance, the image generationunit 201 may first adjust the display mode for the icon 300A byselecting a display position corresponding to a displacement quantityamounting to, for instance, 75% of the displacement quantityrepresenting the full distance between the icon 300A1 and the icon300A0, and then may switch to the second display mode. As a furtheralternative, the image generation unit 201 may first adjust the displaymode for the icon 300A by selecting a display position corresponding toa displacement quantity amounting to, for instance, 125% of thedisplacement quantity representing the full distance between the icon300A1 and the icon 300A0, and then may switch to the second displaymode.

As a further alternative, the image generation unit 201, havingdisplayed the icon 300A2 in the second display mode through the secondprocessing, may then display the icon 300A1 in the first display mode,switch the display mode so as to bring up the icon 300A0 in the initialdisplay and further switch to the second display mode. In this case, theimage generation unit 201 may bring up the icon 300A2 in FIG. 29(c)while the icon 300A1 shown in FIG. 29(c) is on display and may adjustthe display mode for the icon 300A1 on display so as to match that forthe icon 300A2.

As a result, the user is made aware that the operation performed afterthe first processing has not been much different from the operationperformed during the first processing and accordingly is led to performan operation at the position at which the detection reference 40 islocated in the midair image operation mode during or after the secondprocessing.

The processing executed in the first calibration processing mode by thedisplay device 1 in the second embodiment will be explained in referenceto the flowchart presented in FIG. 30 through FIG. 32. Since theprocessing executed in step S801 through step S804 in the flowchartpresented in FIG. 30 is identical to the processing executed in step S1through step S4 in the flowchart presented in FIG. 10, a repeatedexplanation is not provided. In addition, the processing executed instep S805 and step S806 is identical to the processing executed in stepS25 and step S27 in the flowchart presented in FIG. 12, and accordingly,a repeated explanation is not provided.

In step S807, the display position of the icon 300A is altered to switchto the first display mode through the first processing and then theoperation proceeds to step S808 in FIG. 31. The processing executed instep S808 through step S810 in FIG. 31 is identical to the processingexecuted in step S804 through step S806 in FIG. 30. In step S811, thesize of the icon 300A is altered to switch to the second display modethrough the second processing, before the operation proceeds to stepS815.

If a negative decision is made in step S809, processing similar to thatexecuted in step S810 is executed in step S812. In step S813, processingsimilar to that executed in step S30 in the flowchart presented in FIG.12 is executed. Upon making an affirmative decision in step S813, theoperation proceeds to step S815, whereas upon making a negative decisionin step S813, the operation proceeds to step S816. In step S816, thedisplay position of the icon 300A is altered to switch to the seconddisplay mode and then the operation proceeds to step S815. Theprocessing executed in step S815 and S816 is similar to that executed instep S7 and step S8 in FIG. 10. The processing executed subsequently issimilar to the processing executed in step S10 and subsequent steps, asshown in FIG. 6, and is not included in the chart in FIG. 31.

If a negative decision is made in step S805 in FIG. 30, the operationproceeds to the step S817 to execute processing similar to that executedin step S806. In step S818, the display position of the icon 300A isaltered to switch to the first display mode before the operationproceeds to step S819 in FIG. 32. The flowchart presented in FIG. 32 isidentical to the flowchart presented in FIG. 31 except for theprocessing executed in step S822 and step S825, and accordingly, arepeated explanation is not provided. In step S822, the display positionof the icon 300A is altered to switch to the second display mode. Instep S825, the size of the icon 300A is altered to switch to the seconddisplay mode. It is to be noted that the processing executed followingstep S827, which is similar to the processing executed in step S10 andsubsequent steps, as shown in FIG. 6 is not included in FIG. 32. Inaddition, a decision may be made in step S826 as to whether or not theuser has performed an operation to exit the first calibration processingmode. In this case, if the user has performed an operation to exit thefirst calibration processing mode, the operation proceeds to step S827.If, on the other hand, the user has not performed an operation to exitthe first calibration processing mode, the operation may return to stepS804 in FIG. 30 to detect a descending movement of the user's finger.

It is to be noted that while the processing is executed in the firstcalibration processing mode in the explanation provided above, it mayinstead be executed in the second calibration mode. In such a case, theprocessing in step S804 in FIG. 30 through step S814 in FIG. 31 and theprocessing through step S826 in FIG. 32 will be executed instead of theprocessing executed in step S24 through step S32 in the flowchartpresented in FIG. 12 pertaining to the first embodiment. However, if anaffirmative decision is made in step S809 in FIG. 31, display switchoverprocessing is first executed as in step S26 shown in FIG. 12, for theicon 30A before the operation proceeds to step S810 in FIG. 31. Inaddition, if an affirmative decision is made in step S813 in FIG. 31,display switchover processing is likewise executed as in step S31 inFIG. 12 for the icon 30A. In the processing executed as shown in FIG.32, too, upon making an affirmative decision in step S820, displayswitchover is first executed as in step S26 in FIG. 12 for the icon 30Abefore the operation proceeds to step S821 in FIG. 32. If an affirmativedecision is made in step S824 in FIG. 32, display switchover processingis executed as in step S31 in FIG. 12 for the icon 30A.

The image generation unit 201 in the second embodiment first switchesthe display mode for the icon 30A or 300A (first processing) andsubsequently, if the operation detector 13 does not detect a useroperation, the image generation unit 201 adjusts the display to create auser perception as if the icon 30A or 300A has moved along a directionopposite from the direction in which the user has perceived as if theicon 30A or 300A has moved as the display was adjusted through the firstprocessing. As a result, the user is made aware that he has perceived asif the icon 30A or 300A has moved to an excessive extent and thus isexpected to adjust the operating position at which he performs asubsequent press-down operation so that the reach position or thespecified position can be detected at the detection reference 40.

In addition, the image generation unit 201 in the second embodimentfirst switches to the first display mode by increasing the size of theicon 30A or 300A and then switches to the second display mode byreducing the size of the icon 30A or 300A. As a result, the user is madeaware that he has had a perception as if the icon 30A or 300A has movedto an excessive extent during the first processing.

In addition, the image generation unit 201 in the second embodimentfirst switches to the first display mode by moving the display positionof the icon 30A or 300A and then switches the display mode for the icon30A or 300A to the second display mode by moving the display position soas to create a user perception as if it has moved along a directionopposite the direction in which it has been perceived to have movedthrough the switchover to the first display mode. As a result, the useris made aware that he has had a perception as if the icon 30A or 300Ahas moved to an excessive extent during the first processing.

Moreover, the image generation unit 201 in the second embodimentswitches the display mode for the icon 300A1 to the second display modedifferent from the first display mode if the user operation, performedwith respect to the icon 300A1 displayed in the adjusted display mode,i.e., the first display mode, through the first processing, is notdetected at the detection reference 40. Thus, the user is made awarethat he has had a perception as if the icon 300A has remained stationaryand is thus expected to adjust the operating position at which heperforms a subsequent press-down operation to enable detection of thefurthest reach position or the specified position at the detectionreference 40.

It is to be noted that while the display device 1 in the secondembodiment described above includes at least the control unit 20, thedisplay unit 11 and the operation detector 13, the present invention mayinstead be adopted in a control device configured with the control unit20 alone or a control device configured with the control unit 20 and theoperation detector 13. In addition, the control unit 20 only needs toinclude, at least, the calibration unit 203 and the image generationunit 201. A structural element among the structural elements describedearlier may be added as needed in order to realize the variousadvantages described in reference to the second embodiment. In addition,the control device described above may be built into any of varioustypes of electronic devices adopting the second embodiment.

Furthermore, the present invention may be adopted in a detection deviceconfigured with the control unit 20 alone or a detection deviceconfigured with the control unit 20 and the operation detector 13.Moreover, the control unit 20 only needs to include at least thecalibration unit 203 and the image generation unit 201. In order toenable such a detection device to achieve the various advantagesdescribed in reference to the second embodiment, a structural elementamong the structural elements described earlier may be added into thedetection device as deemed necessary.

Third Embodiment

In reference to drawings, a display device 1 in the third embodimentwill be described. The third embodiment will be described in referenceto an example in which the display device 1 in the embodiment is mountedin an operation panel. It is to be noted that the display device 1 inthe embodiment does not need to be mounted in an operation panel and mayinstead be mounted in any type of electronic apparatus as described inreference to the first embodiment and the variations thereof.

The display device 1 in the third embodiment may adopt a structuresimilar to that of the display device 1 in variation 9 of the firstembodiment shown in FIG. 26. It is to be noted that the display device 1in the third embodiment may detect the reach position or the specifiedposition and adjust the display mode for a midair image 30 by adoptingany of the various methods described in reference to the firstembodiment, the variations of the first embodiment and the secondembodiment.

The image generation unit 201 in the display device 1 in the thirdembodiment adjusts the display mode for an icon 30A or 300A based upon auser condition during calibration processing. The display device 1 inthe third embodiment determines a user condition such as whether or notthe user is looking at a midair image 30, and the image generation unit201 adjusts the display mode for the icon 30A or 300A if the user is notlooking at the midair image 30.

The following is an explanation of the calibration processing executedin the embodiment in the second calibration processing mode.

An image capturing device 18 captures an image of the user operating theicon 30A in the midair image 30. An image analysis unit 210 analyzes thecaptured image obtained by photographing the user via the imagecapturing device 18, and makes a decision as to whether or not the useris currently looking at the midair image 30 by determining theorientation of the user's face or body based upon the analysis results.If the image analysis unit 210 decides that the user is not currentlylooking at the midair image 30, the image generation unit 201 adjuststhe display mode for the icon 30A. It is to be noted that in thisembodiment, too, the detection reference control unit 204 calculates thedistance between the reach position or the specified position and thedetection reference 40 and the image generation unit 201 adjusts thedisplay mode for the icon 30A based upon the calculated distance in thesame way as that described in reference to the first embodiment, thevariations thereof and the second embodiment. The user may find itvisually disturbing if the display mode for the icon 30A is adjustedthrough calibration processing while he is looking at it. Accordingly,the image generation unit 201 in the display device 1 in the embodimentadjusts the display mode for the icon 30A while the user is looking awayfrom the midair image 30 and since the user does not witness the displaymode switchover, he does not experience the visual disturbance.

The display device 1 may include a line-of-sight detector that detectsthe line-of-sight of the user, in place of or in addition to the imagecapturing device 18, and may make a decision based upon a detectionoutput provided from the line-of-sight detector as to whether or not theuser is looking at the midair image 30. Based upon the decision-makingresults, the image generation unit 201 adjusts the display mode for theicon 30A while the user is not looking at the midair image 30.

In reference to the flowchart presented in FIG. 33, the calibrationprocessing executed by the display device 1 in the third embodiment willbe explained. Since the processing executed in the flowchart presentedin FIG. 33 is identical to that presented in FIG. 12 except for theprocessing executed in step S858 and step S863, a repeated explanationis not provided. If the reach position is determined in step S857, adecision is made in step S858 as to whether or not the user is lookingat the midair image 30. If it is decided that the user is not currentlylooking at the midair image 30 based upon the orientation of the user'sface or body, indicated in image capturing data generated by the imagecapturing device 18, a negative decision is made in step S858 and theoperation proceeds to step S859. If, on the other hand, it is decidedthat the user is looking at the midair image 30, an affirmative decisionis made in step S858 and the operation goes into standby in step S858.In addition, if a negative decision is made in step S862, processingsimilar to that in step S858 is executed in step S863.

It is to be noted that while the calibration processing is executed inthe second calibration processing mode in the example explained above,it may instead be executed in the first calibration processing mode. Insuch a case, the processing in step S858 in the flowchart presented inFIG. 33 will be executed after making an affirmative decision in step S5in the flowchart presented in FIG. 10 in reference to which the firstembodiment has been explained.

In addition, the image-capturing device 18 or the line-of-sight detectordescribed above does not need to be installed in the display device 1.The image-capturing device 18 may be installed outside the displaydevice 1 and may transmit image-capturing data to the display device 1through wireless communication or via a cable. In addition, theline-of-sight detector may be installed outside the display device 1 andmay transmit the line-of-sight detection results to the display device 1through wireless communication or via a cable.

It is to be further noted that the image generation unit 201 adjusts thedisplay mode for the icon 30A when it is decided that the user is notlooking at the midair image 30 in the example explained above inreference to the third embodiment. Instead, the image generation unit201 may adjust the display mode for the icon 30A when it is decided thatthe user is looking at the midair image 30. In the latter case, theuser, looking at the midair image 30, is able to sense the extent towhich the display mode for the icon 30A is adjusted by the imagegeneration unit 201. Thus, the user can be led to alter the operatingposition if necessary.

In addition, while control is executed in the display device 1 so as toadjust the display mode for the icon 30A when the user is not looking atthe midair image 30 in the explanation provided above, control mayinstead be executed so as to adjust the display mode for the icon 30Abased upon a value indicated in user biometric information. The user'spulse rate indicates a user condition, and may be obtained as such userbiometric information. The user's pulse rate may be obtained via, forinstance, a pulse rate counting device that the user puts on beforestarting to operate the display device 1. Then, the image generationunit 201 may execute control so as to adjust the display mode for theicon 30A when the user's pulse rate increases. The user's pulse rate mayrise when the user, being unable to perform an operation smoothly,becomes frustrated. In this situation, adjustment of the display modefor the icon 30A at the display device 1 will assure better ease ofoperation for the user.

It is to be also noted that if the display mode for the icon 30A isadjusted upon detecting that the user has stopped looking at the midairimage 30, the user may repeatedly perform an operation at the displayposition of the midair image 30 a plurality of times before he stopslooking at the midair image 30. In such a case, the image generationunit 201 will adjust the display mode for the icon 30A based upon theresults of the plurality of operations repeatedly performed, when theuser is no longer looking at the midair image 30. For instance, theimage generation unit 201 may adjust the display mode for the icon 30Abased upon an average value representing the plurality of reachpositions or specified positions, e.g., the arithmetic mean or thegeometric mean of the plurality of reach positions or specifiedpositions, a median of the plurality of reach positions 50, or the mostrecent reach position or specified position among the plurality of reachpositions or specified positions.

The image generation unit 201 in the third embodiment adjusts thedisplay mode for the icon 30A based upon a user condition. As a result,the sense of visual disturbance that the user may experience as thedisplay mode for the icon 30A is altered can be reduced.

In addition, the image generation unit 201 in the third embodimentadjusts the display mode for the icon 30A while the user is not lookingat the midair image 30. As a result, the sense of visual disturbancethat the user may experience as the display mode for the icon 30A isaltered can be reduced.

Furthermore, a decision is made as to whether or not the user is lookingat the midair image 30 in the third embodiment based upon at least oneof; information pertaining to the user's line-of-sight, informationpertaining to the orientation of the user's face and informationpertaining to the orientation of the user's body, obtained throughdetection. Thus, the display mode for the icon 30A can be adjusted whilethe user is not looking at the midair image 30.

(Variation 1 of the third embodiment) A display device 1 in variation 1of the third embodiment will be explained next.

The display device 1 in variation 1 may adopt a structure similar tothat of the display device 1 in the first embodiment shown in FIG. 1 andFIG. 2. It is to be noted that the display device 1 in the thirdembodiment may detect the reach position or the specified position andmay adjust the display mode for a midair image 30 by adopting any of thevarious methods described in reference to the numerous variations of thefirst embodiment and in reference to the second embodiment.

The display device 1 in variation 1 starts display mode adjustment for amidair image 30 in response to a user operation during calibrationprocessing. In this variation, after the user operation ends, the imagegeneration unit 201 starts display mode adjustment for the midair image30. The following is an explanation given in reference to the secondcalibration processing mode.

A decision as to whether or not the display position of the midair imageis to be altered is made based upon the results of a decision as towhether or not a user operation has ended. Accordingly, the control unit20 decides that the user operation has ended once the reach position orthe specified position is determined. The decision as to whether or notthe user operation has ended is made by determining the reach positionor the specified position during the processing executed in step S857and step S860 in the flowchart presented in FIG. 33 in reference towhich the third embodiment has been described. In addition, theprocessing in step S858 and step S863 in the flowchart presented in FIG.33 is not executed.

It is to be noted that once the reach position or the specified positionis determined, the user operation may be determined to have ended, upondetecting a specific gesture signaling display position adjustment (suchas the user switching from the “paper” sign to the “stone” sign). As analternative, once the reach position or the specified position isdetermined, the user operation may be determined to have ended upondetecting that a display position adjustment button brought up ondisplay in the midair image, has been pressed-down by the user. In thiscase, the decision as to whether or not the user operation has ended maybe made instead of executing the processing in step S858 and step S863in the flowchart presented in FIG. 33.

It is to be noted that while the calibration processing is executed inthe second calibration processing mode in the example explained above,it may instead be executed in the first calibration processing mode. Insuch a case, a decision as to whether or not the user operation hasended will be made, after making an affirmative decision in step S5 inthe flowchart presented in FIG. 10 in reference to which the firstembodiment has been explained, and the operation will proceed to step S6upon deciding that the user operation has ended.

In variation 1 of the third embodiment, the image generation unit 201adjusts the display mode for the midair image 30 after the useroperation ends. Thus any sense of visual disturbance that the user mayexperience when the display mode for the midair image 30 is adjusted canbe lessened.

(Variation 2 of the Third Embodiment)

A display device 1 in variation 2 of the third embodiment will beexplained next. The display device 1 in variation 2 may adopt astructure similar to that of the display device 1 in the firstembodiment shown in FIG. 1 and FIG. 2. It is to be noted that thedisplay device 1 in the third embodiment may detect the reach positionor the specified position and may adjust the display mode for the midairimage 30 by adopting any of the various methods described in referenceto the numerous variations of the first embodiment and in reference tothe second embodiment.

The following is an explanation given in reference to the secondcalibration processing embodiment.

When the image generation unit 201 in the display device 1 in variation2 adjusts the display mode for an icon 30A so that it fades out and thenfades back in during the period of time elapsing between the displaymode adjustment start and the display mode adjustment end. Namely, atthe start of the display mode adjustment for an icon 30A the imagegeneration unit 201 gradually lowers the display luminance and thengradually raises the display luminance toward the end of the displaymode adjustment. The user may find it visually disturbing to see thedisplay mode for the icon 30A adjusted by the image generation unit 201as he looks at it during calibration processing. Accordingly, the imagegeneration unit 201 gradually lowers the display luminance as thedisplay mode adjustment for an icon 30A begins. As a result, throughdisplay mode adjustment for the icon 30A is rendered less visible to theuser to reduce the visual disturbance that may otherwise be experiencedby the user.

The image generation unit 201 may lower the display luminance or thecontrast of the icon 30A, may flash the display of the icon 30A atlowered luminance or contrast, or may even hide the display of the icon30A. By rendering the display mode adjustment for the icon 30A, executedby the image generation unit 201 less noticeable, i.e., less visible,through these measures, the user experience will be improved.

As an alternative, the image generation unit 201 may render the icon 30Amore noticeable while the display mode for the icon 30A is adjusted. Theicon 30A may be rendered more noticeable by the image generation unit201 by raising the display luminance or the contrast of the icon 30A, orby flashing the display of the icon 30A while the display mode for theicon 30A is being adjusted. The image generation unit 201 renders thedisplay mode for the icon 30A undergoing adjustment more noticeable byrendering the icon 30A itself more noticeable while the display mode forthe icon 30A is being adjusted so that the user's attention will befocused on the icon 30A itself rather than on the display modeadjustment for the icon 30A. Under such circumstances, the user will notbe distracted by the display mode adjustment of the icon 30A.

The display of the icon 30A is controlled as described above as part ofthe processing executed in step S859 or step S864 in the flowchartpresented in FIG. 33 during adjustment of the display mode the icon 30A.

It is to be noted that the user may be offered a choice of whether ornot to render the midair image movement more noticeable as an option.

Furthermore, while the calibration processing is underway, the displaymode adjustment for the icon 30A executed by the image generation unit201 may be rendered more noticeable so as to ensure that the user isaware of the display mode adjustment for the icon 30A. The imagegeneration unit 201 may raise the display luminance or the contrast ofthe icon 30A, or may flash the display of the icon 30A while the displaymode adjustment for the icon 30A is underway. In contrast to the exampleexplained earlier in which the display mode adjustment for the icon 30Ais rendered less noticeable, the adjustment of the display mode for theicon 30A is rendered more noticeable to the user so that the user canrecognize the display mode for the icon 30A resulting from theadjustment with better clarity.

It is to be noted that the display luminance of the icon 30A is alteredas described above as part of the processing executed in step S859 orstep S864 in the flowchart presented in FIG. 33 during adjustment of thedisplay mode for the icon 30A.

It is also to be noted that while the calibration processing is executedin the second calibration processing mode in the example explainedabove, it may instead be executed in the first calibration processingmode.

In variation 2 of the third embodiment, the display mode for the icon30A is adjusted by altering at least either the luminance of the icon30A or the contrast of the icon 30A. As a result, the sense of visualdisturbance the user may experience as the display mode for the icon 30Ais adjusted can be reduced.

(Variation 3 of the Third Embodiment)

A display device 1 in variation 3 of the third embodiment will beexplained next. The display device 1 in variation 3 may adopt astructure similar to that of the display device 1 in the firstembodiment shown in FIG. 1 and FIG. 2. It is to be noted that thedisplay device 1 in the third embodiment may detect the reach positionor the specified position and may adjust the display mode for a midairimage 30 by adopting any of the various methods described in referenceto the numerous variations of the first mode and in reference to thesecond embodiment.

The following is an explanation given in reference to the secondcalibration processing mode.

The display device 1 in variation 3 adjusts the display mode byreflecting the results of calibration processing, executed based upon auser operation performed with respect to the display position of amidair image 30 corresponding to a given operation screen, in a midairimage 30 corresponding to the next screen brought up on display as achangeover from the given operation screen occurs. In more specificterms, as a user operation is performed with respect to a menu icondisplayed in a midair image 30, a plurality of application icons, viawhich various types of application programs (e.g., music, email and SNSapps) are started up, are brought up on display in a midair image 30.When the display is switched to include these application icons, thedisplay mode of the menu icon is adjusted.

FIG. 34 presents examples of midair images 30 that may be brought up ondisplay in variation 3. FIG. 34(a) shows a midair image 30 provided as afirst display corresponding to an operation screen that includes a menuicon 30A1 and reference lines 310 through 314 (hereafter referred to asa first midair image). The image generation unit 201 displays the menuicon 30A1 in the first midair image 30 by sustaining the initial displaystate with respect to its size and display position.

FIG. 34(b) and FIG. 34(c) each show a midair image 30 provided as asecond display (hereafter referred to as a second midair image) thatcorresponds to an operation screen that includes application icons 30A2,30A3 and 30A4 and the reference lines 310 through 314. FIG. 34(b) showsthe image in the initial display prior to any adjustment in the displaymode for the application icons 30A2 through 30A4. FIG. 34(c) shows theimage following adjustment in the display mode for the application icons30A2 through 30A4. The reference lines 310 through 314 in the first andsecond midair images 30 shown in FIGS. 34(a) through 34(c) are similarto those described in reference to the first embodiment, the variationsthereof, the second embodiment and the variations thereof. In addition,FIG. 34(c) shows a second midair image 30 brought up on display when thefingertip F of the user has been detected at a position above thedetection reference 40 (toward the Z direction+ side).

The user performs an operation by moving his fingertip F downward towardthe menu icon 30A1 in the first midair image 30 brought up on display bythe display control unit 202. If the operation detector 13 detects thereach position or the specified position at the detection reference 40(see FIG. 7(c) pertaining to the first embodiment), the display controlunit 202 switches the display from the first midair image 30 to a secondmidair image 30. In other words, if the operation detector 13 does notdetect the reach position or the specified position at the detectionreference 40 (see FIG. 7(b) or FIG. 7(d) pertaining to the firstembodiment), the display control unit 202 does not switch the displayfrom the first midair image 30 to a second midair image 30.

After the display device 1 is started up, if the operation detector 13detects the reach position or the specified position at the detectionreference 40 following a first operation performed by the user withrespect to the menu icon 30A1, the display control unit 202 switches thedisplay to the second midair image 30 shown in FIG. 34(b). Since theuser operation for the menu icon 30A1 in the first midair image 30 hasbeen performed at the detection reference 40, the user is expected toperform an operation at the detection reference 40 again when theapplication icons 30A2 through 30A4 are brought up in the second midairimage 30 in the initial display as well. For this reason, thecalibration unit 203 does not execute any calibration processing inresponse to the user operation.

If there has been an operation that was not detected at the detectionreference 40 before the operation detector 13 detects the reach positionor the specified position at the detection reference 40, the displaycontrol unit 202 switches the display to the second midair image 30shown in FIG. 34(c). In this situation, before the display control unit202 switches the display to the second midair image 30, the calibrationunit 203 executes calibration processing based upon the operation thathas not been detected at the detection reference 40. In the calibrationprocessing, the detection reference control unit 204 individuallycalculates displacement quantities and sizes for the icons 30A2 through30A4 based upon the distance between the reach position or the specifiedposition and the detection reference 40, as has been explained inreference to the first embodiment, the variations thereof and the secondembodiment and variations thereof.

The image generation unit 201 calculates displacement quantities andsizes for the icons 30A2 through 30A4 based upon the results obtained bydetecting a user operation of moving his fingertip F downward, performedimmediately before (i.e., in the immediately preceding session) theoperation detector 13 detects the reach position or the specifiedposition at the detection reference 40. The distance between the reachposition or the specified position and the detection reference 40,calculated each time the user performs an operation of moving hisfingertip F downward, is stored into the storage unit 205. It is to benoted that the distance between the reach position or the specifiedposition and the detection reference 40 stored in the storage unit 205may be saved by overwriting the previous value with the most recentvalue. The image generation unit 201 individually calculatesdisplacement quantities and sizes for the icons 30A2 through 30A4 byreferencing the distance stored in the storage unit 205.

It is to be noted that if the user performs an operation in which hemoves his fingertip F downward a plurality of times before the operationdetector 13 detects the reach position or the specified position at thedetection reference 40, the image generation unit 201 may calculatedisplacement quantities and sizes for the icons 30A2 through 30A4 asdescribed below. Namely, the image generation unit 201 may individuallycalculate displacement quantities and sizes for the icons 30A2 through30A4 by using the average value of the values each representing thedistance between the reach position or the specified position and thedetection reference 40 calculated in correspondence to one of theplurality of operations.

In addition, the image generation unit 201 may calculate displacementquantities and sizes for the icons 30A2 through 30A4 by incorporatingthe results of an operation is detected by the operation detector 13.For instance, the user may perform a first operation at a position 10 cmabove the detection reference 40 (toward the Z direction+ side) andperform a second operation at the detection reference 40. In thissituation, the image generation unit 201 may calculate the distancebetween the reach position or the specified position and the detectionreference 40 as the average value of the value representing the firstoperation results and the value representing the second operationresults, i.e., 5 cm above.

It is to be noted that if the display device 1 is a device that isbasically used exclusively by a single user, such as a portabletelephone, the image generation unit 201 may calculate the distancebetween the reach position or the specified position and the detectionreference 40 based upon a record of a plurality of operations havingbeen performed by the single user. However, if the display device 1 is asignage system or the like that is likely to be operated by a number ofpeople, the operation record will be deleted each time the displaydevice is operated by a new person. The control unit 20 may photograph,via the image capturing device 18, the face of the user each time thedisplay device 1 is operated, and decide that a change of operator hasoccurred if the face of the user has changed, as has been explained inreference to variation 10 of the first embodiment.

Based upon the displacement quantities and sizes calculated for theicons 30A2 through 30A4, the image generation unit 201 individuallyadjusts the display mode for each of the icons 30A2 through 30A4 therebygenerating a display image corresponding to the second midair image 30.Based upon the display image thus generated, the display control unit202 brings up on display the second midair image 30 shown in FIG. 34(c).

As shown in FIG. 34(c), the icons 30A2 through 30A4 in the second midairimage 30 coming up after the calibration processing are displayed in adisplay mode switched from that for the icons 30A2 through 30A4 in theinitial display. It is to be noted that while the icons 30A2′ through30A4′ in the initial display are indicated by dotted lines superimposedon the second midair image 30 so as to facilitate the explanation, thesecond midair image 30 in FIG. 34(c) does not actually include the icons30A2′ through 30A4′ in the initial display.

The icons 30A2 through 30A4 in the second midair image 30 assume asmaller size than the icons 30A2′ through 30A4′ in the initial displayindicated by the dotted lines, with their display positions alteredalong a direction running toward the vanishing point. As a result, theuser perceives as if the icons 30A2 through 30A4 have moved further awayfrom the user relative to the positions taken when the user hasperformed an operation for the first midair image 30 shown in FIG.34(a). Accordingly, the user is expected to perform a press-downoperation toward a position further downward for the icons 30A2 through30A4 in the second midair image 30 than the position at which the userhas performed the operation for the icon 30A1.

It is to be noted that if the first operation is performed at a positionfurther downward relative to the detection reference, the display modefor the icons 30A2 through 30A4 in the second midair image is adjustedso as to create a user perception as if they have moved closer to theuser. Namely, the size of the icons 30A2 through 30A4 is increased andtheir display positions are altered along a direction running away fromthe vanishing point.

In reference to the flowchart presented in FIG. 35, the calibrationprocessing executed in the display device 1 in variation 3 of the thirdembodiment will be explained.

The processing executed in step S890 and step S891 is similar to thatexecuted in step S21 and step S22 in FIG. 12. In step S892, the firstmidair image (see FIG. 34(a)) that includes the menu icon 30A1 isbrought up on display and the operation proceeds to step S893. In stepS893, processing similar to that in step S24 in FIG. 12 is executed andin step S894 through step S895, processing similar to that in step S29through step S30 is executed. In step S896, to which the operationproceeds after making a negative decision in step S895 (when the reachposition or the specified position is not in alignment with the positionof the detection reference 40), data indicating the distance between thereach position or the specified position and the detection reference 40having been calculated, are stored into the storage unit 205, and theoperation returns to step S893.

Upon making an affirmative decision in step S895 (when the reachposition or the specified position is in alignment with the position ofthe detection reference 40), a decision is made in step S897 as towhether or not data indicating the calculated distance has been storedinto the storage unit 205 in step S896. If data indicating the distanceare stored in the storage unit 205, an affirmative decision is made instep S897 and the operation proceeds to step S898. In step S898, thedisplay is switched to bring up a second midair image (see FIG. 34(c))that includes the application icons 30A2 through 30A4 displayed in adisplay mode adjusted based upon the data indicating the distance, whichare stored in the storage unit 205, and then the processing ends. Ifdata indicating the distance are not stored in the storage unit 205, anegative decision is made in step S897 and the operation proceeds tostep S899. In step S899, the display is switched to bring up a secondmidair image (see FIG. 34(b)) that includes the application icons 30A2through 30A4 displayed in the initial display mode, and then theprocessing ends.

It is to be noted that a specific effect may be added when the displaycontrol unit 202 switches the display from the first midair image 30 tothe second midair image 30. For instance, the display control unit 202may execute processing for fading out and fading in the icons byadjusting the icon luminance. Namely, as the display switchover from thefirst midair image 30 starts, the display control unit 202 may graduallylower the luminance of the first midair image 30 and as the switchoverto the second midair image 30 nears completion, it may graduallyincrease the luminance. As a result, the change in the icon display modewill be rendered less noticeable to the user while the display isswitched to the second midair image 30 and thus, the user will notexperience as much visual disturbance during the display modeswitchover.

In variation 3 of the third embodiment, if the position at which anoperation has been performed with respect to the menu icon 30A1 in thefirst midair image 30 is not detected at the detection reference 40, thecontrol unit 20 executes processing corresponding to the menu icon 30A1.At this time, the image generation unit 201 displays a second midairimage 30, brought up after the first midair image 30, by adjusting thedisplay mode for the application icons 30A2 through 30A4 based upon theoperation performed for the menu icon 30A1 in the first midair image 30.As a result, the display mode for the icons 30A2 through 30A4 isadjusted while the display is switched from the first midair image 30 tothe second midair image 30, which makes it possible to ensure that theuser does not experience a visual disturbance as the display mode isswitched.

In addition, if the position at which a user operation has beenperformed with respect to the first midair image 30 is not detected atthe detection reference 40, the control unit 20 does not switch thedisplay to a second midair image 30 in variation 3 of the thirdembodiment. As a result, the user is made aware that the operation hasnot been performed at the position at which the detection reference 40is set.

(Variation 4 of the Third Embodiment)

A display device 1 in variation 4 of the third embodiment will beexplained next. The display device 1 in variation 4 may adopt astructure similar to that of the display device 1 in the firstembodiment shown in FIG. 1 and FIG. 2.

The feature distinguishing variation 4 of the third embodiment fromvariation 3 of the third embodiment described above will be explainedbelow. Even when the operation detector 13 does not detect the reachposition or the specified position pertaining to a specific icon 30A1included in the first midair image 30 brought up in the first display,at the detection reference 40, the display control unit 202 switches toa second display by bringing up a second midair image 30. It is to benoted that the specific icon 30A1 may be an icon operated when thedisplay device is engaged in operation, among a plurality of iconsstored in the display device 1, such as a start button operated to startoperation at the display device 1 or a lock-release button operated tocancel a lock applied on an operation in the display device 1. Namely,the specific icon 30A1 is an icon corresponding to an operation that theuser is expected to continuously perform on a midair image 30 even afterthe user has performed an operation with respect to the icon 30A1.

The processing executed when the operation detector 13 has detected afirst operation performed by the user at the detection reference 40 willbe explained. In this case, the image generation unit 201 generatesdisplay image data corresponding to a second midair image 30 thatincludes icons 30A2 through 30A4 in the initial display state, as shownin FIG. 34(b), as in variation 3 of the third embodiment. The displaycontrol unit 202 switches the display to the second midair image 30corresponding to the display image data generated by the imagegeneration unit 201.

The processing executed when the reach position or the specifiedposition pertaining to a press-down operation performed by the user withrespect to the menu icon 30A1 in the first midair image 30, has not beendetected at the detection reference 40 by the operation detector 13 willbe explained next. In this situation, the calibration unit 203 executescalibration processing. In the calibration processing, the detectionreference control unit 204 individually calculates displacementquantities and sizes for the icons 30A2 through 30A4 based upon thedistance between the reach position or the specified position and thedetection reference 40, as has been explained in reference to the firstembodiment, the variations thereof and the second embodiment. Based uponthe displacement quantities and the sizes calculated for the icons 30A2through 30A4, the image generation unit 201 individually adjusts thedisplay mode for the icons 30A2 through 30A4, thereby generating adisplay image corresponding to a second midair image 30. The displaycontrol unit 202 brings up on display the second midair image 30 shownin FIG. 34(c) based upon the display image thus generated. In otherwords, if an operation performed for the specific icon 30A1 has not beendetected at the detection reference 40, a second midair image 30 havingundergone the calibration processing, is displayed.

It is to be noted that in variation 4 of the third embodiment, too, aspecific effect may be added when the display control unit 202 switchesthe display from the first midair image 30 to the second midair image30, as in the variation 3. Namely, the display control unit 202 mayexecute processing for fading out and fading in the icons by adjustingthe icon luminance. In this situation, as the display switchover fromthe first midair image 30 starts, the display control unit 202 maygradually lower the luminance of the first midair image 30 and as theswitchover to the second midair image 30 nears completion, it maygradually increase the luminance. As a result, the change in the icondisplay mode will be rendered less noticeable to the user while thedisplay is switched to the second midair image 30 and thus, the userwill not experience a visual disturbance during the display modeswitchover.

It is to be noted that if a user operation, performed on an icon amongthe icons 30A2 through 30A4 in the second midair image 30, i.e., an icondifferent from the specific icon 30A1 in the first display, is notdetected at the detection reference 40, the display device 1 does notexecute any of the processing allocated to the icons 30A2 through 30A4.

In reference to the flowchart presented in FIG. 36, the calibrationprocessing executed in the display device 1 in variation 4 of the thirdembodiment will be explained.

The processing executed in step S900 through step S905 is similar tothat executed in step S890 through step S895 in FIG. 35. If anaffirmative decision is made in step S905 (when the reach position orthe specified position is in alignment with the detection reference 40),the operation proceeds to step S908, in which processing similar to thatin step S899 in FIG. 35 is executed before the processing in theflowchart ends. If, on the other hand, a negative decision is made instep S905 (when the reach position or the specified position is not inalignment with the detection reference 40), the operation proceeds tostep S906, in which processing similar to that in step S896 in FIG. 35is executed and then the operation proceeds to step S907. The processingexecuted in step S907 is similar to that executed in step S898 in FIG.35.

It is to be noted that while data indicating the distance between thereach position and the detection reference are obtained throughcalculation and are stored through the processing executed in step S906,it is not strictly necessary to store the data. In step S907, a secondimage may be brought up on display by adjusting the display mode basedupon the data obtained through calculation without storing the data.

In variation 4 of the third embodiment, if the position at which anoperation performed with respect to a specific icon 30A1 in the firstmidair image 30 is not detected at the detection reference 40, thecontrol unit 20 executes processing corresponding to the menu icon 30A1.Namely, the image generation unit 201 displays a second midair image 30brought up after the first midair image 30, by adjusting the displaymode for the application icons 30A2 through 30A4 based upon theoperation performed for the specific icon 30A1 in the first midair image30. As a result, the display mode for the icons 30A2 through 30A4 isadjusted while the display is switched from the first midair image 30 tothe second midair image 30, which makes it possible to ensure that theuser does not experience visual discomfort as the display mode switches.

In addition, if the position at which a user operation has beenperformed for an icon different from the specific icon 30A1 is notdetected at the detection reference 40, the control unit 20 does notswitch the display to a second midair image 30 in variation 4 of thethird embodiment. As a result, the user is made aware that the operationhas not been performed at the position at which the detection reference40 is set.

In variation 4 of the third embodiment, the display is switched to asecond midair image 30 even when an operation performed at the displayposition at which the specific icon 30A1, constituted with a buttonoperated to release a lock on the display device 1, is displayed is notdetected at the detection reference 40. When the user performs anoperation for releasing a lock on the display device 1, the user ishighly likely to continue to perform an operation on the midair image30. In variation 4 of the third embodiment, the user is able to performan operation on the midair image 30 having undergone the calibrationprocessing without having to start up the calibration processing modeagain after the lock on the display device 1 is released and thus,better ease of operation is assured.

While the specific icon 30A1 is constituted with a lock-release buttonin the example described above, the present invention is not limited tothis example. For instance, the specific icon 30A1 may be a start buttonoperated when starting operation in the display device 1. In such acase, the user does not need to perform any superfluous operations whenstarting operation in the display device 1. In addition, sincecalibration processing is executed at the start of operation in thedisplay device 1, a midair image 30 in the midair image operation modeis brought up in the adjusted display mode. Since this eliminates theneed for the user to start up the calibration processing mode again,better ease of operation is assured.

In addition, the specific icon 30A1 may be a menu button brought up ondisplay after the start button has been operated or a calibration icon300A such as that shown in FIG. 4.

It is to be noted that while the display device 1 in the thirdembodiment and variations 1 through 4 thereof described above includesat least the control unit 20, the display unit 11 and the operationdetector 13, the present invention may instead be adopted in a controldevice configured with the control unit 20 alone or a control deviceconfigured with the control unit 20 and the operation detector 13. Inaddition, the control unit 20 only needs to include, at least, thecalibration unit 203 and the image generation unit 201. A structuralelement among the structural elements described earlier may be added asneeded in order to realize the various advantages described in referenceto the third embodiment and variations 1 through 4 thereof. In addition,the control device described above may be built into any of varioustypes of electronic devices adopting the third embodiment and variationsthereof.

Furthermore, the present invention may be adopted in a detection deviceconfigured with the control unit 20 alone or a detection deviceconfigured with the control unit 20 and the operation detector 13.Moreover, the control unit 20 only needs to include at least thecalibration unit 203 and the image generation unit 201. In order toenable such a detection device to achieve the various advantagesdescribed in reference to the third embodiment and variations 1 through4 thereof, a structural element among the structural elements describedearlier may be added into the detection device as deemed necessary.

Fourth Embodiment

In the various embodiments and variations thereof described above, amidair image is generated via the image-forming optical system 12 incorrespondence to an image displayed at the display unit 11 in thedisplay device 1. However, a midair image may be generated by adopting astructure other than that described above through the following methods.The structures explained below simply represent examples and a midairimage may be generated by assuming a structure other than thosedescribed below.

A midair image may be generated at a display device by bringing up animage to be viewed with the right eye and an image to be viewed with theleft eye, which has parallactic offset relative to the image to beviewed with the right eye, on display at the display unit of the displaydevice so as to create an image perceived by the user to have depth,unlike the images displayed at the display unit. In the display devicein this embodiment, calibration processing is executed that is differentfrom the calibration processing explained in reference to the firstembodiment, the variations thereof, the second embodiment, the thirdembodiment and the variations thereof in that a user perception iscreated as if the depth of an image has changed by adjusting the extentof parallactic offset for the parallax image. Namely, the display devicein the embodiment adjusts the extent of parallactic offset for theparallax image based upon the distance between the reach position or thespecified position determined based upon a user operation and thedetection reference, detected in the same way as in the firstembodiment, the variations thereof, the second embodiment, the thirdembodiment and the variations thereof described earlier. As a result,the user's perception of the position of a midair image 30 along thedepthwise direction changes and thus, adjustment can be made so that thereach position or the specified position determined based upon the useroperation is set at the detection reference.

In reference to drawings, a display device 1 in the fourth embodimentwill be described. The fourth embodiment will be described in referenceto an example in which the display device 1 in the embodiment is mountedin an operation panel. It is to be noted that the display device 1 inthe embodiment does not need to be mounted in an operation panel andmay, instead, be mounted in any type of electronic apparatus, as hasbeen described in reference to the first through the third embodimentsand the variations thereof.

FIG. 37 shows the display device 1 in the fourth embodiment in anexploded perspective. The display device 1 in the fourth embodimentincludes, for instance, a lenticular lens 16 of the known art, disposedabove (toward the Z direction+ side) the display unit 11, in place of orin addition to the image-forming optical system 12 included in thedisplay device 1 in the first embodiment shown in FIG. 1. Via thelenticular lens 16, a right-eye image and a left-eye image displayed atthe display unit 11 are viewed with the user's right eye and left eyerespectively and as a result, the user perceives the parallax imagedisplayed at the display unit 11 as a stereoscopic image (midair image30) having depth. The display device 1 further includes a detector 15that detects the user's position (the position at which the user isstanding) while he is looking at the image on display at the displayunit 11.

It is to be noted that the display device 1 may instead include a slitused to generate a parallax image through a parallax barrier system orthe like of the known art, formed at a position above the display unit(toward the Z direction+ side), instead of the lenticular lens 16.Namely, the display device 1 may adopt any of various modes that enabledisplay of an image having parallax discernible with the naked eye. Thedisplay device 1 structured as described above creates a user perceptionas if the depthwise position of the midair image 30 has changed by thebinocular parallax effect. In this sense, the display device 1 in thefourth embodiment is different from the first embodiment and thevariations thereof, the second embodiment and the third embodiment andthe variations thereof, in which the depthwise position of an icon 30Aor 300A is altered by using a monocular depth cue.

It is to be noted that the display device 1 may adopt a structure thatdisplays a parallax image that can be viewed by the user wearing specialeyeglasses. In such a case, the display device 1 and the specialeyeglasses may be controlled through, for instance, the active shuttermethod so as to create a user perception that the parallax image he islooking at is a midair image 30. The display device 1 provides a displayat the display unit 11 through high-speed switching between theright-eye image and the left-eye image. The user puts on specialeyeglasses such as eyeglasses with a liquid crystal shutter system andlooks at the parallax image displayed at the display unit 11 through theeyeglasses. At this time, the display at the display unit 11 and theshutter open/close states are controlled in synchronization so thatwhile the right-eye image is up on display at the display unit 11 in thedisplay device 1, the left-eye shutter in the special glasses is closedand while the left-eye image is up on display at the display unit 11 inthe display device 1, the right-eye shutter is closed in the specialeyeglasses.

FIG. 38 is a block diagram of the essential structure adopted in thedisplay device 1 in the fourth embodiment. In addition to the structuralelements in the block diagram for the first embodiment provided in FIG.2, the display device 1 in this embodiment includes a detector 15 thatdetects the line-of-sight, the eye ball movement or the like of the userlooking at the parallax image on display at the display device 1. Thedetector 15 may be, for instance, an image capturing device capable ofeye tracking of the known art that detects eye movement in the user byradiating infrared light toward the eyeballs of the user looking at theparallax image and detecting light reflected from the eyeballs.

In addition, the control unit 20 has a function realized in the form ofa user decision-making unit 221 that makes a decision, based upon thedetection results provided from the detector 15, as to whether or notthe line-of-sight of the user or the position (standing position), atwhich the user is looking at the parallax image, is optimal for creatinga user perception that the parallax image is a midair image 30.

It is to be noted that the display device 1 may adopt a structureachieved by adding the detector 15 and the user decision-making unit 221to any of the structures shown in the block diagrams in reference towhich the variations of the first embodiment, the second embodiment, thethird embodiment and the variations thereof have been explained.

When a parallax image created by taking advantage of the binocularparallax effect is perceived as a midair image 30 by the user inconjunction with the display device 1 in the embodiment, a region wherethe user is able to view the parallax image as a stereoscopic image anda pseudoscopic region are present in an alternate pattern. This meansthat depending upon where the user is standing, the user, looking at theparallax image with the naked eye may perceive it as a distorted midairimage 30 or may not be able to perceive the midair image 30 as astereoscopic image. In order to prevent such an occurrence, the positiontaken by the user of the display device 1 is determined via the imagecapturing device 18 or a detector 15 and the user decision-making unit221, and the display device 1 guides the user to an optimal standingposition for perceiving the parallax image as a midair image 30 basedupon the decision-making results.

The user decision-making unit 221 detects the standing position taken bythe user looking at the parallax image brought up on display by thedisplay device 1 based upon image capturing data generated by the imagecapturing device 18. The user decision-making unit 221 decides that theuser is able to perceive the midair image 30 in an optimal manner if theuser's standing position having been detected is within an area wherethe parallax image appears as a stereoscopic image. If, on the otherhand, the detected standing position is in a pseudoscopic region, theuser decision-making unit 221 decides that the user is unable toperceive the midair image 30 in an optimal manner, i.e., the standingposition taken by the user is not desirable. If the user decision-makingunit 221 decides that the standing position taken by the user is notdesirable, the display control unit 202 brings up a message such as“Step to the right” over the midair image 30 by controlling, forinstance, the display unit 11 so as to guide the user to an area where astereoscopic view is afforded. It is to be noted that instead ofdisplaying a message on the midair image 30, the display device 1 mayinclude, for instance, a speaker or the like to lead the user to adjusthis position through an audio message, a warning sound or the like.

It is to be noted that while the user decision-making unit 221 makes adecision with respect to the standing position taken by the user basedupon the image capturing data provided from the image capturing device18 in the example explained above, the control described above may beexecuted when the line-of-sight of the user, detected via the detector15, is within a pseudoscopic region, instead.

In addition, instead of leading the user to adjust his standing positionwith a message brought up on display, an audio message or the likeprovided by the display device 1, the display position at which theparallax image is displayed may be adjusted so as to move a stereoscopicview-enabled region to the user's standing position or the user'sline-of-sight position. In such a case, the display control unit 202will adjust the display position at which the display image data aredisplayed on the display unit 11 relative to the lenticular lens 16. Asan alternative, the image generation unit 201 may generate display imagedata having undergone processing of the known art, which are to be usedto correct distortion or the like of the midair image 30 attributable tothe current standing position of the user and the display control unit202 may bring up on display the display image data resulting from thecorrection as the parallax image. As a further alternative, the displaydevice 1 may include a drive device that moves the lenticular lens 16over the XY plane so as to position the lenticular lens 16 relative tothe display unit 11.

While the user of the display device 1 in the fourth embodimentperceives the parallax image displayed at the display unit 11 as amidair image 30, any change in the depth of the midair image 30 isattributable to the user's perception and thus, a change in the depth ofthe midair image 30 is perceived differently from one user to another.This means that some users may not be able to perceive any change in thedepth of the midair image 30. In order to ensure that this does nothappen, the display device 1 in the embodiment executes calibrationprocessing so as to make it possible for a user to perceive a change inthe depth of the midair image 30 if the particular user has not beenable to perceive a change in the depth of the midair image 30. It is tobe noted that while the following explanation is given in reference tothe first calibration mode, the calibration processing may be adopted inthe second calibration processing mode, as well.

While a midair image 300 brought up on display during the calibrationprocessing at the display device 1 in the fourth embodiment includes anicon 300A, it does not include any reference lines 310 through 314,unlike the midair image 300 brought up on display in the firstembodiment, as shown in FIG. 4.

If the reach position or the specified position with respect to apress-down operation performed by the user is detected, via theoperation detector 13, to be located further upward (toward the Zdirection+ side) relative to the detection reference 40 (see FIG. 7(b)),the image generation unit 201 generates display image data expressing aparallax image in which the icon 300A appears to be located further awayfrom the user. The image generation unit 201 generates display imagedata expressing a parallax image having a less pronounced parallaxeffect than the parallax image constituting the midair image 300 in theinitial display. The image generation unit 201 determines the degree ofparallax effect to manifest in the parallax image based upon thedistance between the reach position or the specified position and thedetection reference 40. Data indicating parallax quantities eachrepresenting a degree of parallax effect correlated a specific distancebetween the reach position or the specified position and the detectionreference 40, obtained in advance based upon results of testing and thelike, are stored in the storage unit 205. The image generation unit 201selects a parallax quantity by referencing the data stored in thestorage unit 205. Through these measures, a user perception as if theicon 300A is located further away from the user (toward the Z direction− side) than the icon 300A in the midair image 300 in the initialdisplay is created.

It is to be noted that if the reach position or the specified positionwith respect to a press-down operation performed by the user isdetected, via the operation detector 13, to be located further downward(toward the Z direction − side) relative to the detection reference 40(see FIG. 7(d)), the image generation unit 201 generates display imagedata expressing a parallax image in which the icon 300A appears to belocated closer the user. The image generation unit 201 generates displayimage data expressing a parallax image manifesting a more pronouncedparallax effect than the extent of parallax in the parallax imageconstituting the midair image 300 in the initial display. The imagegeneration unit 201 determines the degree of parallax effect to manifestin the parallax image based upon the distance between the reach positionor the specified position and the detection reference 40 by referencingthe data stored in advance in the storage unit 205. Through thesemeasures, a user perception as if the icon 300A is located closer to theuser (toward the Z direction+ side) than the icon 300A in the midairimage 300 in the initial display is created.

Once the position at which the icon 300A is displayed in midair isadjusted as described above, the user decision-making unit 221 makes adecision as to whether or not the user perceives a change in the depthof the icon 300A. In this step, the user decision-making unit 221 makesa decision based upon information pertaining to the user's eyes. Thedecision-making processing executed by the user decision-making unit 221will be explained next.

The detector 15 described earlier detects the distance between the leftpupil and the right pupil of the user viewing the parallax image as theinformation pertaining to the user's eyes. The user decision-making unit221 calculates the position at which the user perceives the icon 300Aalong the Z direction based upon the display position at which the icon300A is displayed in midair (i.e., its position along the Z direction)in correspondence to the parallax image displayed at the display unit 11and the distance between the left pupil and the right pupil detected viathe detector 15. If the position of the icon 300A along the Z directionin the parallax image and the position of the icon 300A along the Zdirection as perceived by the user are offset relative to each other byan extent greater than a predetermined offset quantity, the userdecision-making unit 221 decides that the user has not perceived anychange in the depth of the icon 300A.

If the user decision-making unit 221 decides that the user has notperceived a change in the depth of the icon 300A, the image generationunit 201 further adjusts the parallax quantity for the parallax image.In this situation, the image generation unit 201 further adjusts theparallax quantity for the parallax image based upon the extent of offsetbetween the position of the icon 300A along the Z direction in theparallax image and the position having been calculated, at which theuser perceives the icon 300A to be located along the Z direction.Namely, if the image generation unit 201 has been engaged in processingfor reducing the degree of parallax effect in the parallax image, itgenerates a parallax image with less parallax. Through these measures, auser perception as if the midair image 300 has moved even further awayfrom the user is created. If, on the other hand, the image generationunit 201 has been engaged in processing for increasing the degree ofparallax effect in the parallax image, it generates a parallax imagewith more parallax. Through these measures, a user perception as if themidair image 300 has moved even closer to the user is created.

The calibration processing executed in the display device 1 in thefourth embodiment will be explained in reference to the flowchartpresented in FIG. 39. It is to be noted that the flowchart in FIG. 39only includes step S910 through step S921 and does not includesubsequent steps. The processing executed in step S921 and subsequentsteps is similar to the processing executed in step S7 and subsequentsteps in the flowchart presented in FIG. 10.

The processing executed in step S910 through step S912 is similar to theprocessing executed in step S1 through step S3 in the flowchartpresented in FIG. 10 pertaining to the first embodiment. In step S913, adecision is made with respect to the standing position taken by the userlooking at the midair image 300. If it is decided, based upon the imagecapturing data generated by the image capturing device 18, that thestanding position taken by the user is within a stereoscopicviewing-enabled region where the parallax image can be viewed as astereoscopic image, an affirmative decision is made in step S913 and theoperation proceeds to step S914. If, on the other hand, the standingposition taken by the user is determined to be within a pseudoscopicregion, a negative decision is made in step S913 and the operation waitsin standby. In this case, a message leading the user to adjust thestanding position may be brought up on display or an audio messageleading a positional adjustment may be output, as explained earlier.

The processing executed in step S914 and step S915 is similar to theprocessing in step S4 and step S5 in the flowchart presented in FIG. 10.In step S916, a decision is made as to whether or not the reach positionis in alignment with the position of the detection reference 40. If thereach position is in alignment with the position of the detectionreference 40, an affirmative decision is made in step S916 and theoperation proceeds to step S921, which will be described later. If, onthe other hand, the reach position is not in alignment with the positionof the detection reference 40, a negative decision is made in step S916and the operation proceeds to step S917. In step S917, the imagegeneration unit 201 adjusts the display mode for the icon 300A byadjusting the parallax quantity for the parallax image and then theoperation proceeds to step S918.

In step S918, the line-of-sight of the user is detected based upon thedetection output provided from the detector 15, and then the operationproceeds to step S919. In step S919, a decision is made as to whether ornot the user perceives a midair position taken by the icon 300Afollowing the display mode adjustment. If the user perceives the midairposition of the icon 300A, i.e., if the extent of offset between theposition of the icon 300A along the Z direction in the parallax imageand the position of the icon 300A along the Z direction as perceived bythe user is equal to or less than a predetermined offset quantity, anaffirmative decision is made in step S919 and the operation proceeds tostep S921. If the user does not perceive the midair position of the icon300A, i.e., if the extent of offset between the position of the icon300A along the Z direction in the parallax image and the position of theicon 300A along the Z direction as perceived by the user is greater thanthe predetermined offset quantity, a negative decision is made in stepS919 and the operation proceeds to step S920. In step S920, the imagegeneration unit 201 adjusts the degree of parallax effect for theparallax image based upon the specific extent of offset between positionof the icon 300A along the Z direction in the parallax image and theposition of the icon 300A along the Z direction as perceived by theuser, and the display mode for the icon 300A is thus adjusted before theoperation proceeds to step S921. In step S921, the operation exits thefirst calibration processing mode.

It is to be noted that the processing may be executed in the secondcalibration processing mode by executing the processing in step S913 inthe flowchart presented in FIG. 39 following the processing in step S23in the flowchart presented in FIG. 12 pertaining to the firstembodiment. In addition, the processing in step S918 through step S920in the flowchart presented in FIG. 39 will be executed after theprocessing executed in step S28 and step S32.

It is to be noted that the user decision-making unit 221 may make adecision as to whether or not the user perceives the icon 300A basedupon the distance between the left pupil and the right pupil of hiseyes. In such a case, the distance between the left pupil and the rightpupil in the user's eyes will be obtained in advance via the detector 15and be stored as distance data in the storage unit 205. If the parallaxquantity in the parallax image corresponding to the icon 300A in theinitial display is adjusted by the image generation unit 201 throughcalibration processing, the user decision-making unit 221 will obtainthe direction of the user's left and right eye movement and the extentof the eye movement based upon a detection output from the detector 15.If an eye movement has occurred along a direction corresponding to thedirection in which the change in the depth of the icon 300A has occurredas a result of the adjustment in the parallax quantity in the parallaximage and the extent of the eye movement is within a predeterminedrange, the user decision-making unit 221 will decide that the user hasperceived the change having occurred along the direction of the depth ofthe icon 300A.

In addition, if the user perceives that a change has occurred along thedirection of depth of a midair image 30, the diameter of the user'spupils, too, normally changes with the change having occurred in thedirection of the depth of the midair image 30.

Accordingly, the detector 15 may detect the pupil diameter in the user'seyes as information pertaining to the user's eyes and the userdecision-making unit 221 may make a decision based upon the detectedpupil diameter as to whether or not the user has perceived a change inthe direction of the depth of the icon 300A. In this situation, the userdecision-making unit 221 will make a decision based upon the directionof along which the pupil diameter has changed following the adjustmentof the parallax quantity in the parallax image (whether the pupildiameter has increased or decreased) relative to the pupil diametermeasured before the adjustment of the parallax quantity in the parallaximage and the extent to which the pupil diameter has changed. Namely, ifthe pupil diameter has changed in a direction corresponding to thedepthwise direction along which the change has occurred with respect tothe icon 300A and the extent of the change is within a predeterminedrange, the user decision-making unit 221 will decide that the user hasperceived a change having occurred in the depthwise direction withrespect to the icon 300A.

It is to be noted that regardless of whether the display device 1displays a parallax image as a midair image 30 by adopting a naked eyemethod or an eyeglass method, the user cannot see the midair image 30 asa stereoscopic image if one of his eyes is closed. Accordingly, if thedetector 15 detects that one of the user's eyes is closed, the displaydevice 1 may lead the user to open both eyes. The display control unit202 may bring up on display a message such as “Please open both eyes andlook at the midair image” in the midair image 30, or the display device1 may include a speaker or the like so as to enable the control unit 20to output the message above as an audio message or the like.

The image control unit 202 in the fourth embodiment brings up on displaytwo parallactic images having parallactic offset at the display unit 11and controls the display of the midair image 30 based upon the standingposition taken by the user. Through these measures, it is ensured thatthe user takes a position in an area where he can view the midair image30 as a stereoscopic image.

The image generation unit 201 in the fourth embodiment displays twoparallactic images having parallactic offset at the display unit 11 anddisplays a midair image 30 based upon information related to theposition at which a user operation has been detected and informationpertaining to the user's eyes. As a result, it is ensured that the useris able to see the midair image 30 as a stereoscopic image throughexploitation of the binocular parallax effect.

In addition, the image generation unit 201 in the fourth embodimentadjusts the parallax quantity, i.e. the extent of parallactic offsetmanifested by two parallax images, based upon the distance between theleft pupil and the right pupil in the user's eyes or the pupil diameter.As a result, if the user has not perceived any change with respect tothe depth of the midair image 30, a further change can be made withrespect to the depth of the midair image 30 so that the user is able toperceive the spatial position of the midair image 30.

(Variation 1 of the Fourth Embodiment)

A display device 1 in variation 1 of the fourth embodiment adjusts thedisplay position of a midair image along the depthwise direction byexecuting calibration processing through a method different from themethod in which the parallax quantity representing the degree ofparallax effect in a parallax image is adjusted.

In an example of such a variation, the display unit 11 is allowed tomove along the Z direction in the display device 1 as explained below.

FIG. 40 is a block diagram of the essential structure adopted in thedisplay device 1. The display device 1 in variation 1 includes a displayposition adjustment unit 500 and a display position control unit 220 inaddition to the structural elements in the display device 1 in thefourth embodiment, as shown in FIG. 37 and FIG. 38.

The display position adjustment unit 500, which includes a drive unitsuch as a motor or an actuator, moves the display unit 11 along theoptical axis of the lenticular lens 16, as indicated by the arrow, so asto adjust the position at which the midair image 30 is perceived bymoving the position along the Z direction, i.e., along the optical axis.In order to move the midair image 30 upward, i.e., further away from thedisplay unit 11, the display position adjustment unit 500 moves thedisplay unit 11 upward, i.e., closer to the lenticular lens 16. In orderto move the midair image 30 downward, i.e., closer to the display unit11, the display position adjustment unit 500 moves the display unit 11downward, i.e., away from the lenticular lens 16. It is to be noted thatinstead of moving the display unit 11, the display position adjustmentunit 500 may move the lenticular lens 16 along the Z axis or may moveboth the display unit 11 and the lenticular lens 16 along the Z axis.

The display position adjustment unit 500 in the display device 1 invariation 1 of the fourth embodiment moves the display unit 11 along theZ axis if the user still does not perceive that the midair image 30 hasmoved along the depthwise direction even after the parallax quantity hasbeen adjusted for a parallax image through the calibration processing.The display position adjustment unit 500 determines the extent to whichthe display unit 11 is to move and the direction along which the displayunit 11 is to move based upon an offset quantity indicating the extentof offset between the position of the icon 300A along the Z direction inthe parallax image and the calculated position at which by the userperceives the icon 300A to be located along the Z direction. Namely, ifthe image generation unit 201 has executed processing for reducing theparallax quantity for the parallax image, the display positionadjustment unit 500 moves the display unit 11 toward the Z direction −side. As a result, a user perception as if the midair image 300 hasmoved even further away from the user is created. If the imagegeneration unit 201 has executed processing for increasing the parallaxquantity for the parallax image, the display position adjustment unit500 moves the display unit 11 toward the Z direction+ side. As a result,a user perception as if the midair image 300 has moved even closer tothe user is created.

In the display device 1 in variation 1 of the fourth embodiment, theprocessing in step S920 in the flowchart presented in FIG. 39 isexecuted by the display position adjustment unit 500 by adjusting thedisplay unit 11.

It is to be noted that the display device 1 may lead the user to openboth eyes if one of them is closed in the same way as that described inreference to the fourth embodiment. In addition, the display positionadjustment unit 500 may move the display unit 11 under suchcircumstances.

In variation 1 of the fourth embodiment, the position of the midairimage 30 is adjusted through adjustment of the position of the displayunit 11. Through these measures, it is ensured that the user is able toperceive the display position of the midair image 30 in space.

Is to be noted that instead of moving the position of the display unit11 via the display position adjustment unit 500, the technology taughtin International Publication No. 2011/158911 may be adopted in thedisplay unit 11. Namely, the display unit 11 may adopt a structure thatenables light field display of a 3D stereoscopic image and a midairimage 30 may be formed at various positions in midair along the opticalaxis by bringing up on display an image for two-dimensional display atthe display unit 11.

FIG. 41 schematically illustrates a display unit 11 and an image-formingoptical system 12B that may be included in such a structure in asectional view taken over the ZX plane. The image-forming optical system12B is disposed at the display surface of the display unit 11. Theimage-forming optical system 12B includes a microlens array 123constituted with a plurality of microlenses 122 disposed in atwo-dimensional pattern. The microlenses 122 are each disposed incorrespondence to a plurality of display pixels P at the display unit11. It is to be noted that while a single microlens 122 is disposed incorrespondence to 5×5 display pixels P in the example presented in FIG.41 in order to simplify the illustration, each microlens 122 is actuallydisposed in correspondence to a greater number of display pixels P. Themicrolens array 123 is disposed at a position set apart from the displaysurface of the display unit 11 by a distance matching the focal length fof the microlenses 122 toward the Z direction+ side. Each microlens 122projects light traveling from display pixels P onto a specific imageplane along the Z direction based upon the image on display. It is to benoted that lenticular lenses may be used in place of the microlenses122.

In order for the various light points LP that compose a midair image 30to be formed in space, the light forming a given light point LP isemitted from some of the display pixels P, each covered by one of aplurality of different microlenses 122 at the display unit 11. It is tobe noted that the light point LP, which is an image displayed in midairvia the display unit 11 and the microlenses 122, is a midair image. Inthe example presented in FIG. 41, light emitted from the shaded displaypixels P is projected via the corresponding microlenses 122 to form thelight point LP. In this situation, the display pixels P to emit thelight that will form the light point LP are assigned corresponding tothe plurality of different microlenses 122 in a number matching thenumber of display pixels P covered by a single microlens 122 (5×5 in theexample presented in FIG. 41). Based upon the pattern in which thesedisplay pixels P are assigned, the position of a light point LP formedin midair along the Z direction can be adjusted. The midair image 30 iscomposed of light points P thus formed. The position of the midair image30 along the Z direction can be thus adjusted by altering the imagedisplayed at the display unit 11. It is to be noted that the position ofthe midair image 30 along the Z direction may be adjusted through acomputer-generated hologram (CGH) technology by constituting the displaydevice 1 with a DMD or the like.

(Variation 2 of the Fourth Embodiment)

The display device in variation 2 of the fourth embodiment may adopt astructure similar to that of the display device in variation 1 of thefourth embodiment shown in FIG. 40. Namely, the display device 1 invariation 2 of the fourth embodiment includes a structural element thatmakes it possible for the display unit 11 to move. In the display device1 in variation 2, control is executed to move a midair image 30 alongthe depthwise direction either by moving the display unit 11 or byadjusting the parallax quantity in a parallax image in correspondence tothe extent to which the midair image 30 is to move along the depthwisedirection.

Under normal circumstances, a human being is able to perceive thedistance by which the depthwise position of a midair image 30 is alteredby adjusting the parallax quantity in a parallax image more accuratelythan the distance by which the midair image 30 actually moves throughspace. Accordingly, in calibration processing executed in the displaydevice 1 in variation 2 of the fourth embodiment, the display positionadjustment unit 500 moves the display unit 11 if the extent to which themidair image 30 is to move exceeds a predetermined value and the imagegeneration unit 201 adjusts the parallax quantity for the parallax imageif the extent to which the midair image 30 is to move is equal to orless than the predetermined value.

The calibration processing executed in the display device 1 in variation2 of the fourth embodiment will be explained in reference to theflowchart presented in FIG. 42. It is to be noted that the flowchart inFIG. 42 only includes step S930 through step S941 and does not includesubsequent steps. The processing executed in step S941 and subsequentsteps is similar to the processing executed in step S7 and subsequentsteps in the flowchart presented in FIG. 10.

The processing executed in step S930 through step S936 is similar to theprocessing executed in step S910 through step S916 in the flowchartpresented in FIG. 39 pertaining to the fourth embodiment. In step S937,to which the operation proceeds after making a negative decision in stepS936, a displacement quantity indicating the extent to which the midairimage 30 is to move is calculated based upon the distance between thereach position and the detection reference 40 and then the operationproceeds to step past 938. In step S938, a decision is made as towhether or not the displacement quantity calculated for the midair image30 exceeds a predetermined value. If it is decided that the displacementquantity exceeds the predetermined value, an affirmative decision ismade in step S938 and the operation proceeds to step S939. In step S939,the display position adjustment unit 500 moves the display unit 11, andthe operation proceeds to step S941. If, on the other hand, thedisplacement quantity is equal to or less than the predetermined value,a negative decision is made in step S938 and the operation proceeds tostep S940. In step S940, the image generation unit 201 adjusts thedisplay mode for the icon 30A before the operation proceeds to stepS941. In step S921, the processing in the first calibration processingmode ends.

It is to be noted that the processing may be executed in the secondcalibration processing mode by executing the processing in step S937through step S940 in the flowchart presented in FIG. 42 instead of theprocessing in step S26 through step S28 in the flowchart presented inFIG. 12 pertaining to the first embodiment. In addition, the processingin step S937 through step S940 in the flowchart presented in FIG. 42will be executed instead of the processing executed in step S29, stepS30 and step S32 in FIG. 12.

It is to be noted that the display device 1 may switch to the control bymoving the display unit 11 or the control by adjusting the parallaxquantity for a parallax image in correspondence to the extent to which amidair image 30 is to move when moving the midair image 30 based upon auser operation in the midair image operation mode as well as in thecalibration processing mode. For instance, if the user performs anoperation for moving the midair image 30 by 30 cm, the display positionadjustment unit 500 moves the display unit 11. If, on the other hand,the user performs an operation for moving the midair image by 1 mm, theimage generation unit 201 adjusts the parallax quantity for the parallaximage.

In addition, the control executed in the display device 1 in variation 2of the fourth embodiment may be adopted in the first embodiment and thevariations thereof, the second embodiment, the third embodiment and thevariations thereof. Namely, when the extent to which a midair image 30or 300 is to move exceeds a predetermined value, the display positionadjustment unit 500 may move the display unit 11 and if the extent towhich the midair image 30 or 300 is to move is equal to or less than thepredetermined value, the image generation unit 201 may adjust thedisplay mode for an icon 30A or 300A.

It is to be noted that while the display device 1 in the fourthembodiment and variations 1 and 2 thereof described above includes atleast the control unit 20, the display unit 11 and the operationdetector 13, the present invention may instead be adopted in a controldevice configured with the control unit 20 alone or a control deviceconfigured with the control unit 20 and the operation detector 13. Inaddition, the control unit 20 only needs to include, at least, thecalibration unit 203 and the image generation unit 201. A structuralelement among the structural elements described earlier may be added asneeded in order to realize the various advantages described in referenceto the fourth embodiment and variations 1 and 2 thereof. In addition,the control device described above may be built into any of varioustypes of electronic devices adopting the fourth embodiment and thevariations thereof.

Furthermore, the present invention may be adopted in a detection deviceconfigured with the control unit 20 alone or a detection deviceconfigured with the control unit 20 and the operation detector 13.Moreover, the control unit 20 only needs to include at least thecalibration unit 203 and the image generation unit 201. In order toenable such a detection device to realize the various advantagesdescribed in reference to the fourth embodiment and variations 1 and 2thereof, a structural element among the structural elements describedearlier may be added into the detection device as deemed necessary.

In all the embodiments and variations described above, a midair imagemay be generated by condensing laser light in midair and forming plasmawith air molecules so as to emit light in midair. Through this method, athree-dimensional image is generated as a real image in midair bycontrolling the laser light condensing position at any desired positionin the three-dimensional space. In another midair image generationmethod, an image may be generated in midair via a display device havinga function of creating fog in the air in addition to a projectorfunction by creating a screen with the fog in the air and projecting animage onto the screen formed with the fog (fog display).

A program enabling the various types of processing to be executed at thedisplay device 1 to move the position of a midair image 30 may berecorded into a computer-readable recording medium, and the calibrationmay be executed based upon the program read into a computer system. Itis to be noted that the “computer system” in this context may include anOS (operating system) and hardware such as peripheral devices.

It is to be also noted that the “computer system” may include a homepageprovider environment (or a display environment) in conjunction with theWWW system. In addition, the “computer-readable recording medium” may bea non-volatile writable memory such as a flexible disk, amagneto-optical disk, a ROM or a flash memory, a portable medium such asa CD-ROM, or a storage device such as a hard disk built into a computersystem. Furthermore, the “computer-readable recording medium” may be astorage medium capable of holding a program over a specific length oftime, such as a volatile memory (e.g., DRAM (dynamic random accessmemory)) in a computer system functioning as a server or a client whenthe program is transmitted via a communication network such as theInternet or via a communication line such as a telephone line.

The “program” stored in a storage device or the like in a computersystem may be transmitted to another computer system via a transmissionmedium or on a transmission wave in a transmission medium. The“transmission medium” through which the program is transmitted in thiscontext refers to a medium having a function of informationtransmission, examples of which include a network (communicationnetwork) such as the Internet and a communication line such as atelephone line. The program described above may enable only some of thefunctions described earlier. Furthermore, the program may be adifferential file (differential program) that works in conjunction witha program already recorded in the computer system so as to enable thefunctions described earlier.

As long as the features characterizing the present invention remainintact, the present invention is in no way limited to the particulars ofthe embodiments described above and other modes or combinations that areconceivable within the technical teaching of the present invention arealso within the scope of the invention.

The disclosure of the following priority application is hereinincorporated by reference:

-   Japanese Patent Application No. 2016-128209 filed Jun. 28, 2016

REFERENCE SIGNS LIST

1 . . . display device, 11 . . . display unit, 12 . . . image-formingoptical system, 13 . . . operation detector, 14 . . . sound collector,15 . . . detector, 18 . . . image capturing device, 20 . . . controlunit, 103 . . . display unit, 201 . . . image generation unit, 202 . . .display control unit, 203 . . . calibration unit, 204 . . . detectionreference control unit, 206 . . . velocity·acceleration detection unit,207 . . . reach position predicting unit, 208 . . . sound detectionunit, 209 . . . index mark display control unit, 210 . . . imageanalysis unit, 220 . . . display position control unit, 221 . . . userdecision-making unit, 500 . . . display position adjustment unit

1. A display device, comprising: a display unit that displays, at aremote position, a display image that includes a first image and asecond image; an operation detection unit that detects an operation by auser on the display image; and a display control unit that adjusts adisplay mode for at least one of the first image and the second imagebased upon the operation detected by the operation detection unit. 2.The display device according to claim 1, wherein: the operationdetection unit detects a position of the operation by the user relativeto a position used as a reference to detect the operation by the user.3. The display device according to claim 1, wherein: the operationdetection unit detects whether or not the operation by the user has beenperformed within a predetermined distance from a position used as areference to detect the operation by the user.
 4. The display deviceaccording to claim 3, wherein: the display control unit adjusts thedisplay mode for at least one of the first image and the second image incase that the operation by the user has not been performed within thepredetermined distance from the position used as the reference to detectthe operation by the user.
 5. The display device according to claim 1,wherein: the display control unit adjusts the display mode by alteringat least one of a position and a size of the second image relative tothe first image displayed at the display unit.
 6. The display deviceaccording to claim 1, wherein: the display control unit adjusts thedisplay mode by altering sharpness of at least one of the first imageand the second image displayed at the display unit.
 7. The displaydevice according to claim 1, wherein: the display control unit adjuststhe display mode by adding a shade/shadow to at least one of the firstimage and the second image displayed at the display unit.
 8. (canceled)9. (canceled)
 10. A display device, comprising: a display unit thatdisplays a display image together with reference information displayedby the display device at a position set apart from the display device bya predetermined distance; an operation detection unit that detects anoperation by a user on the display image; an acquisition unit that setsa detection reference near the display image and ascertains a positionalrelationship between the detection reference and the operation by theuser; and a control unit that executes display control in which adisplay mode for the display image in relation to the referenceinformation displayed by the display device is altered based upon thepositional relationship ascertained by the acquisition unit.
 11. Thedisplay device according to claim 10, wherein: the control unit executesdisplay control in which the display mode for the display image inrelation the reference information is altered so as to create a userperception as if a position of the display image has moved closer to theuser or away from the user.
 12. The display device according to claim10, wherein: the control unit executes display control in which adisplay position of the display image, in relation to the referenceinformation displayed by the display device, is altered based upon thepositional relationship ascertained by the acquisition unit.
 13. Thedisplay device according to claim 10, wherein: the control unit executesdisplay control in which a display size of the display image, inrelation to the reference information displayed by the display device,is altered based upon the positional relationship ascertained by theacquisition unit.
 14. The display device according to claim 10, wherein:the reference information displayed by the display device is a referenceline, a shadow image or an index mark.
 15. A control device thatadjusts, based upon an operation by a user with respect to a display ina midair, the display, comprising: an acquisition unit that ascertains apositional relationship between a first reference used for detection ofthe operation and a position at which the operation is detected; and acontrol unit that adjusts the display in relation to a second referenceused as a depth cue for the display based upon the positionalrelationship ascertained by the acquisition unit.
 16. The control deviceaccording to claim 15, wherein: the control unit adjusts the display soas to create a user perception as if a position of the display has movedcloser to the user or away from the user.
 17. The control deviceaccording to claim 16, wherein: the control unit determines a directionalong which the position of the display is to be altered based upon thepositional relationship.
 18. The control device according to claim 16,wherein: the operation is a press-down operation performed as if topress down on the display; and the control unit adjusts the display soas to change the position of the display along a direction determinedbased upon the press-down operation or along a direction opposite fromthe direction determined based upon the press-down operation.